Merge pull request #1474 from jeanas/tool-recommendations

Update tool recommendations

Author: webknjaz

source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst

@@ -1,3 +1,5 @@
+.. _trusted-publishing:
+
 =============================================================================
 Publishing package distribution releases using GitHub Actions CI/CD workflows
 =============================================================================

source/guides/tool-recommendations.rst

@@ -4,111 +4,172 @@
 Tool recommendations
 ====================
 
-If you're familiar with Python packaging and installation, and just want to know
-what tools are currently recommended, then here it is.
+The Python packaging landscape consists of many different tools. For many tasks,
+the :term:`Python Packaging Authority <Python Packaging Authority (PyPA)>`
+(PyPA, the working group which encompasses many packaging tools and
+maintains this guide) purposefully does not make a blanket recommendation; for
+example, the reason there are many build backends is that the landscape was
+opened up in order to enable the development of new backends serving certain users'
+needs better than the previously unique backend, setuptools. This guide does
+point to some tools that are widely recognized, and also makes some
+recommendations of tools that you should *not* use because they are deprecated
+or insecure.
+
+
+Virtual environments
+====================
+
+The standard tools to create and use virtual environments manually are
+:ref:`virtualenv` (PyPA project) and :doc:`venv <python:library/venv>` (part of
+the Python standard library, though missing some features of virtualenv).
+
+
+Installing packages
+===================
+
+:ref:`Pip` is the standard tool to install packages from :term:`PyPI <Python
+Package Index (PyPI)>`. You may want to read pip's recommendations for
+:doc:`secure installs <pip:topics/secure-installs>`. Pip is available by default
+in most Python installations through the standard library package
+:doc:`ensurepip <python:library/ensurepip>`.
+
+Alternatively, consider :ref:`pipx` for the specific use case of installing Python
+applications that are distributed through PyPI and run from the command line.
+Pipx is a wrapper around pip and venv that installs each
+application into a dedicated virtual environment. This avoids conflicts between
+the dependencies of different applications, and also with system-wide applications
+making use of the same Python interpreter (especially on Linux).
+
+For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`.
+
+.. todo:: Write a "pip vs. Conda" comparison, here or in a new discussion.
+
+Do **not** use ``easy_install`` (part of :ref:`setuptools`), which is deprecated
+in favor of pip (see :ref:`pip vs easy_install` for details). Likewise, do
+**not** use ``python setup.py install`` or ``python setup.py develop``, which
+are also deprecated (see :ref:`setup-py-deprecated` for background and
+:ref:`modernize-setup-py-project` for migration advice).
+
+
+Lock files
+==========
+
+:ref:`pip-tools` and :ref:`Pipenv` are two recognized tools to create lock
+files, which contain the exact versions of all packages installed into an
+environment, for reproducibility purposes.
+
+
+Build backends
+==============
+
+.. important::
 
+   Please, remember: this document does not seek to steer the reader towards
+   a particular tool, only to enumerate common tools. Different use cases often
+   need specialized workflows.
 
-Application dependency management
-=================================
+Popular :term:`build backends <build backend>` for pure-Python packages include,
+in alphabetical order:
 
-* Use :ref:`pip` in a `secure manner`_ to install a Python application and its
-  dependencies during deployment.
+- :doc:`Flit-core <flit:pyproject_toml>` -- developed with but separate from :ref:`Flit`.
+  A minimal and opinionated build backend. It does not support plugins.
 
-* Use :ref:`virtualenv` or :doc:`venv <python:library/venv>` to isolate
-  application-specific dependencies from a shared Python installation. [4]_
+- Hatchling_ -- developed with but separate from :ref:`Hatch`. Supports plugins.
 
-* Use `pip-tools`_, :ref:`pipenv`, or `poetry`_ to generate the fully-specified
-  application-specific dependencies, when developing Python applications.
+- PDM-backend_ -- developed with but separate from :ref:`PDM`. Supports plugins.
 
-.. _secure manner: https://pip.pypa.io/en/latest/topics/secure-installs/
-.. _pip-tools: https://github.com/jazzband/pip-tools
-.. _Poetry: https://python-poetry.org/
+- Poetry-core_ -- developed with but separate from :ref:`Poetry`. Supports
+  plugins.
 
-Installation tool recommendations
-=================================
+  Unlike other backends on this list, Poetry-core does not support the standard
+  :ref:`[project] table <writing-pyproject-toml>` (it uses a different format,
+  in the ``[tool.poetry]`` table).
 
-* Use :ref:`pip` to install Python :term:`packages <Distribution Package>` from
-  :term:`PyPI <Python Package Index (PyPI)>`. [1]_ [2]_ Depending on how :ref:`pip`
-  is installed, you may need to also install :ref:`wheel` to get the benefit
-  of wheel caching. [3]_
+- :ref:`setuptools`, which used to be the only build backend. Supports plugins.
 
-* Use :ref:`virtualenv` or :doc:`venv <python:library/venv>` to isolate
-  project-specific dependencies from a shared Python installation. [4]_
+  .. caution::
 
-* If you're looking for management of fully integrated cross-platform software
-  stacks, consider:
+     If you use setuptools, please be aware that some features that predate
+     standardisation efforts are now deprecated and only *temporarily kept*
+     for compatibility.
 
-  * :ref:`buildout`: primarily focused on the web development community
+     In particular, do **not** use direct ``python setup.py`` invocations. On the
+     other hand, configuring setuptools with a :file:`setup.py` file is still fully
+     supported, although it is recommended to use the modern :ref:`[project] table
+     in pyproject.toml <writing-pyproject-toml>` (or :file:`setup.cfg`) whenever possible and keep
+     :file:`setup.py` only if programmatic configuration is needed. See
+     :ref:`setup-py-deprecated`.
 
-  * :ref:`spack`, :ref:`hashdist`, or :ref:`conda`: primarily focused
-    on the scientific community.
+     Other examples of deprecated features you should **not** use include the
+     ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table
+     <pyproject-guide-build-system-table>` in :file:`pyproject.toml` instead), and
+     the ``easy_install`` command (cf. :ref:`pip vs easy_install`).
 
+Do **not** use :ref:`distutils`, which is deprecated, and has been removed from
+the standard library in Python 3.12, although it still remains available from
+setuptools.
 
-Packaging tool recommendations
-==============================
+For packages with :term:`extension modules <extension module>`, it is best to use
+a build system with dedicated support for the language the extension is written in,
+for example:
 
-* Use :ref:`setuptools` to define projects. [5]_ [6]_
+- :ref:`setuptools` -- natively supports C and C++ (with third-party plugins for Go and Rust),
+- :ref:`meson-python` -- C, C++, Fortran, Rust, and other languages supported by Meson,
+- :ref:`scikit-build-core` -- C, C++, Fortran, and other languages supported by CMake,
+- :ref:`maturin` -- Rust, via Cargo.
 
-* Use :ref:`build` to create :term:`Source Distributions
-  <Source Distribution (or "sdist")>` and :term:`wheels <Wheel>`.
 
-If you have binary extensions and want to distribute wheels for multiple
-platforms, use :ref:`cibuildwheel` as part of your CI setup to build
-distributable wheels.
+Building distributions
+======================
 
-* Use `twine <https://pypi.org/project/twine>`_ for uploading distributions
-  to :term:`PyPI <Python Package Index (PyPI)>`.
+The standard tool to build :term:`source distributions <source distribution (or
+"sdist")>` and :term:`wheels <wheel>` for uploading to PyPI is :ref:`build`.  It
+will invoke whichever build backend you :ref:`declared
+<pyproject-guide-build-system-table>` in :file:`pyproject.toml`.
 
+Do **not** use ``python setup.py sdist`` and ``python setup.py bdist_wheel`` for
+this task. All direct invocations of :file:`setup.py` are :ref:`deprecated
+<setup-py-deprecated>`.
 
-Publishing platform migration
-=============================
+If you have :term:`extension modules <extension module>` and want to distribute
+wheels for multiple platforms, use :ref:`cibuildwheel` as part of your CI setup
+to build distributable wheels.
 
-The original Python Package Index implementation (previously hosted at
-`pypi.python.org <https://pypi.python.org>`_) has been phased out in favour
-of an updated implementation hosted at `pypi.org <https://pypi.org>`_.
 
-See :ref:`Migrating to PyPI.org` for more information on the status of the
-migration, and what settings to change in your clients.
+Uploading to PyPI
+=================
 
-----
+For projects hosted on GitHub, it is recommended to use the :ref:`trusted publishing
+<trusted-publishing>`, which allows the package to be securely uploaded to PyPI
+from a GitHub Actions job. (This is not yet supported on software forges other
+than GitHub.)
 
-.. [1] There are some cases where you might choose to use ``easy_install`` (from
-       :ref:`setuptools`), e.g. if you need to install from :term:`Eggs <Egg>`
-       (which pip doesn't support).  For a detailed breakdown, see :ref:`pip vs
-       easy_install`.
+The other available method is to upload the package manually using :ref:`twine`.
 
-.. [2] The acceptance of :pep:`453` means that :ref:`pip`
-       will be available by default in most installations of Python 3.4 or
-       later.  See the :pep:`rationale section <453#rationale>` from :pep:`453`
-       as for why pip was chosen.
+**Never** use ``python setup.py upload`` for this task. In addition to being
+:ref:`deprecated <setup-py-deprecated>`, it is insecure.
 
-.. [3] `get-pip.py <https://github.com/pypa/get-pip/#readme>`_ and
-       :ref:`virtualenv` install
-       :ref:`wheel`, whereas :ref:`ensurepip` and :ref:`venv <venv>` do not
-       currently.  Also, the common "python-pip" package that's found in various
-       linux distros, does not depend on "python-wheel" currently.
 
-.. [4] Beginning with Python 3.4, ``venv`` will create virtualenv environments
-       with ``pip`` installed, thereby making it an equal alternative to
-       :ref:`virtualenv`. However, using :ref:`virtualenv` will still be
-       recommended for users that need cross-version consistency.
+Workflow tools
+==============
 
-.. [5] Although you can use pure :ref:`distutils` for many projects, it does not
-       support defining dependencies on other projects and is missing several
-       convenience utilities for automatically populating distribution metadata
-       correctly that are provided by ``setuptools``. Being outside the
-       standard library, ``setuptools`` also offers a more consistent feature
-       set across different versions of Python, and (unlike ``distutils``),
-       recent versions of ``setuptools`` support all of the modern metadata
-       fields described in :ref:`core-metadata`.
+These tools are environment managers that automatically manage virtual
+environments for a project. They also act as "task runners", allowing you to
+define and invoke tasks such as running tests, compiling documentation,
+regenerating some files, etc. Some of them provide shortcuts for building
+distributions and uploading to PyPI, and some support lock files for
+applications. They often call the tools mentioned above under the hood. In
+alphabetical order:
 
-       Even for projects that do choose to use ``distutils``, when :ref:`pip`
-       installs such projects directly from source (rather than installing
-       from a prebuilt :term:`wheel <Wheel>` file), it will actually build
-       your project using :ref:`setuptools` instead.
+- :ref:`Flit`,
+- :ref:`Hatch`,
+- :doc:`nox <nox:index>`,
+- :ref:`PDM`,
+- :ref:`Pipenv`,
+- :ref:`Poetry`,
+- :doc:`tox <tox:index>`.
 
-.. [6] `distribute`_ (a fork of setuptools) was merged back into
-       :ref:`setuptools` in June 2013, thereby making setuptools the default
-       choice for packaging.
 
-.. _distribute: https://pypi.org/project/distribute
+.. _hatchling: https://pypi.org/project/hatchling/
+.. _pdm-backend: https://backend.pdm-project.org
+.. _poetry-core: https://pypi.org/project/poetry-core/

source/guides/writing-pyproject-toml.rst

@@ -40,6 +40,7 @@ three possible TOML tables in this file.
    :ref:`setup-py-deprecated`.
 
 
+.. _pyproject-guide-build-system-table:
 
 Declaring the build backend
 ===========================