PEP 394: Allow for more flexibility in handling /usr/bin/python

pep-0394.txt

@@ -5,104 +5,167 @@ Last-Modified: $Date$
 Author: Kerrick Staley <[email protected]>,
         Nick Coghlan <[email protected]>,
         Barry Warsaw <[email protected]>,
-        Petr Viktorin <[email protected]>
+        Petr Viktorin <[email protected]>,
+        Miro Hrončok <[email protected]>,
 Status: Active
 Type: Informational
 Content-Type: text/x-rst
 Created: 02-Mar-2011
-Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018
+Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018,
+              12-Apr-2019
 Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html
 
 
 Abstract
 ========
 
 This PEP provides a convention to ensure that Python scripts can continue to
-be portable across ``*nix`` systems, regardless of the default version of the
-Python interpreter (i.e. the version invoked by the ``python`` command).
+be portable across ``*nix`` systems, regardless of the version of the Python
+interpreter invoked by the ``python`` command.
 
 * ``python2`` will refer to some version of Python 2.x.
 * ``python3`` will refer to some version of Python 3.x.
-* for the time being, all distributions *should* ensure that ``python``,
-  if installed, refers to the same target as ``python2``, unless the user
-  deliberately overrides this or a virtual environment is active.
-* however, end users should be aware that ``python`` refers to ``python3``
-  on at least Arch Linux (that change is what prompted the creation of this
-  PEP), so ``python`` should be used in the shebang line only for scripts
-  that are source compatible with both Python 2 and 3.
-* in preparation for an eventual change in the default version of Python,
-  Python 2 only scripts should either be updated to be source compatible
-  with Python 3 or else to use ``python2`` in the shebang line.
-
+* Distributions can choose whether ``python`` is installed and, if it is,
+  whether it refers to the same target as ``python2`` or ``python3``.
+* End users should be aware that the ``python`` command is inconsistent
+  across Unix-like systems.
+* In activated virtual environments, ``python`` always refers to the
+  environment's Python version.
+* Upstream projects should prefer the ``python3`` shebang even if they support
+  both Python 3 and Python 2.
+* Upstream projects only supporting Python 2 should use the ``python2``
+  shebang.
 
 Recommendation
 ==============
 
-* Unix-like software distributions (including systems like Mac OS X and
+For distributors
+----------------
+
+* Unix-like software distributions (including systems like macOS and
   Cygwin) should install the ``python2`` command into the default path
   whenever a version of the Python 2 interpreter is installed, and the same
   for ``python3`` and the Python 3 interpreter.
 * When  invoked, ``python2`` should run some version of the Python 2
   interpreter, and ``python3`` should run some version of the Python 3
   interpreter.
-* If the ``python`` command is installed, it should invoke the same version of
-  Python as the ``python2`` command (however, note that some distributions
-  have already chosen to have ``python`` implement the ``python3``
-  command; see the `Rationale`_ and `Migration Notes`_ below).
-* The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should
-  likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``,
-  with the original commands invoking these versions by default, but possibly
-  invoking the Python 3.x versions instead if configured to do so by the
+* If the ``python`` command is installed, it should either invoke the same
+  version of Python as the ``python3`` command or as the ``python2``
+  command.
+* Distributors may choose to set the behavior of the ``python`` command,
+  not provide it at all, or make it configurable by the user or
   system administrator.
-* In order to tolerate differences across platforms, all new code that needs
+* The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should
+  likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``;
+  Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``.
+  The original commands should either invoke the same version of Python
+  as the ``python`` command, or not be available at all.
+* When packaging software, distributors are encouraged to change less specific
+  shebangs to more specific ones. This ensures software is used with the latest
+  version of Python available, and it can remove a dependency on Python 2.
+  The specifics are left on the distributors.
+  Examples of this include changing ``python`` shebangs to ``python3``
+  when Python 3.x is supported, changing ``python`` shebangs to ``python2``
+  when Python 3.x is not yet supported, or changing ``python3`` shebangs to
+  ``python3.8`` if the software is built with Python 3.8.
+* When a virtual environment (created by the PEP 405 ``venv`` package or a
+  similar tool such as ``virtualenv`` or ``conda``) is active, the ``python``
+  command should refer to the virtual environment's interpreter and should
+  always be available.
+  The ``python3`` or ``python2`` command (according to the environment's
+  interpreter) should also be available.
+
+For developers
+--------------
+
+* In order to tolerate differences across platforms, new code that needs
   to invoke the Python interpreter should not specify ``python``, but rather
-  should specify either ``python2`` or ``python3`` (or the more specific
-  ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_).
+  should specify either ``python3`` or ``python2`` (or the more specific
+  ``python3.x`` and ``python2.x`` versions; see the `Migration Notes`_).
   This distinction should be made in shebangs, when invoking from a shell
   script, when invoking via the system() call, or when invoking in any other
   context.
-* One exception to this is scripts that are deliberately written to be source
-  compatible with both Python 2.x and 3.x. Such scripts may continue to use
-  ``python`` on their shebang line.
-* When packaging software that is source compatible with both versions,
-  distributions may change such ``python`` shebangs to ``python3``.
-  This ensures software is used with the latest version of
-  Python available, and it can remove a dependency on Python 2.
 * When reinvoking the interpreter from a Python script, querying
   ``sys.executable`` to avoid hardcoded assumptions regarding the
   interpreter location remains the preferred approach.
-* In controlled environments aimed at expert users, where being explicit
-  is valued over user experience (for example, in test environments and
-  package build systems), distributions may choose to not provide the
-  ``python`` command even if ``python2`` is available.
-  (All software in such a controlled environment must use ``python3`` or
-  ``python2`` rather than ``python``, which means scripts that deliberately
-  use ``python`` need to be modified for such environments.)
-* When a virtual environment (created by the PEP 405 ``venv`` package or a
-  similar tool) is active, the ``python`` command should refer to the
-  virtual environment's interpreter. In other words, activating a virtual
-  environment counts as deliberate user action to change the default
-  ``python`` interpreter.
+* Scripts that are deliberately written to be source compatible with both
+  Python 3.x and 2.x should use ``python3`` on their shebang line
+  (see `Rationale`_ for details).
+* One exception to this is scripts that are deliberately written to be used
+  on an “*old system*” where ``python3`` (or even ``python2``) is not available
+  (such as the default Python installed on macOS or RHEL 6).
+  Such scripts may continue to use ``python`` on their shebang line.
+* Scripts targeting both “*old systems*” and systems without the default
+  ``python`` command need to make a compromise and document this situation.
+  Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar
+  means) is the recommended workaround for this problem.
+* Applications designed exclusively for a specific environment (such as
+  a container or virtual environment) may continue to use the ``python``
+  command name.
 
 These recommendations are the outcome of the relevant python-dev discussions
 in March and July 2011 ([1]_, [2]_), February 2012 ([4]_),
-September 2014 ([6]_), and discussion on GitHub in April 2018 ([7]_).
+September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_)
+and on python-dev in February 2019 ([8]_).
+
 
 
-Rationale
-=========
+History of this PEP
+===================
 
-This recommendation is needed as, even though the majority of distributions
-still alias the ``python`` command to Python 2, some now alias it to
+In 2011, the majority of distributions
+aliased the ``python`` command to Python 2, but some started switching it to
 Python 3 ([5]_). As some of the former distributions did not provide a
 ``python2`` command by default, there was previously no way for Python 2 code
 (or any code that invokes the Python 2 interpreter directly rather than via
 ``sys.executable``) to reliably run on all Unix-like systems without
 modification, as the ``python`` command would invoke the wrong interpreter
 version on some systems, and the ``python2`` command would fail completely
-on others. The recommendations in this PEP provide a very simple mechanism
+on others. This PEP originally provided a very simple mechanism
 to restore cross-platform support, with minimal additional work required
-on the part of distribution maintainers.
+on the part of distribution maintainers. Simplified, the recommendation was:
+
+1. The ``python`` command was preferred for code compatible with both
+   Python 2 and 3 (since it was available on all systems, even those that
+   already aliased it to Python 3).
+2. The ``python`` command should always invoke Python 2 (to prevent
+   hard-to-diagnose errors when Python 2 code is run on Python 3).
+3. The ``python2`` and ``python3`` commands should be available to specify
+   the version explicitly.
+
+However, these recommendations implicitly assumed that Python 2 would always be
+available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404),
+distributions are making Python 2 optional or removing entirely.
+This means either removing the ``python`` command or switching it to invoke
+Python 3, invalidating respectively the first or second recommendation.
+Also, some distributors decided that their users are better served by
+ignoring the PEP's recommendations, making the PEP's supposedly
+cross-platform recommendations on ``python`` and ``python2`` in shebangs
+increasingly unreliable.
+
+
+.. _rationale:
+
+Current Rationale
+=================
+
+As of 2019, nearly all new systems include Python 3 and the ``python3``
+command. This makes the ``python3`` command the best general choice for
+code that can run on either Python 3.x or 2.x, even though it is not
+available everywhere.
+
+The recommendation is skewed toward current and future systems, leaving
+behind “*old systems*” (like RHEL 6 or default Python installed on macOS).
+On these systems, Python software is rarely updated and any recommendations
+this PEP makes would likely be ignored.
+
+Also, since distributors often ignored recommendations the PEP gave
+regarding the ``python`` command (for what they saw as legitimate special
+needs), this PEP now gives them broad control over the command.
+Correspondingly, users are advised to not use the ``python`` command
+in cross-platform code.
+Instead, this PEP specifies the expected behavior of the ``python3`` and
+``python2`` commands, which is not controversial.
 
 
 Future Changes to this Recommendation
@@ -111,7 +174,7 @@ Future Changes to this Recommendation
 This recommendation will be periodically reviewed over the next few years,
 and updated when the core development team judges it appropriate. As a
 point of reference, regular maintenance releases for the Python 2.7 series
-will continue until at least 2020.
+will continue until January 2020.
 
 
 Migration Notes
@@ -129,27 +192,25 @@ making such a change.
   and other users. Updating the ``python`` command to invoke ``python3``
   by default indicates that a distribution is willing to break such scripts
   with errors that are potentially quite confusing for users that aren't
-  yet familiar with the backwards incompatible changes in Python 3. For
+  familiar with the backwards incompatible changes in Python 3. For
   example, while the change of ``print`` from a statement to a builtin
   function is relatively simple for automated converters to handle, the
-  SyntaxError from attempting to use the Python 2 notation in versions of
-  Python 3 prior to 3.4.2 is thoroughly confusing if you aren't already
-  aware of the change::
+  SyntaxError from attempting to use the Python 2 notation in Python 3
+  may be confusing for users that are not aware of the change::
 
       $ python3 -c 'print "Hello, world!"'
         File "<string>", line 1
           print "Hello, world!"
-                              ^
-      SyntaxError: invalid syntax
-
-  (In Python 3.4.2+, that generic error message has been replaced with the
-  more explicit "SyntaxError: Missing parentheses in call to 'print'")
-* Avoiding breakage of such third party scripts is the key reason this
-  PEP recommends that ``python`` continue to refer to ``python2`` for the
-  time being. Until the conventions described in this PEP are more widely
-  adopted, having ``python`` invoke ``python2`` will remain the recommended
-  option.
-* The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on
+                ^
+      SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello, world!")?
+
+  While this might be obvious for experienced Pythonistas, such scripts
+  might even be run by people who are not familiar with Python at all.
+  Avoiding breakage of such third party scripts was the key reason this
+  PEP used to recommend that ``python`` continue to refer to ``python2``.
+* The error message ``python: command not found`` tends to be surprisingly
+  actionable, even for people unfamiliar with Python.
+* The ``pythonX.X`` (e.g. ``python3.6``) commands exist on modern systems, on
   which they invoke specific minor versions of the Python interpreter. It
   can be useful for distribution-specific packages to take advantage of these
   utilities if they exist, since it will prevent code breakage if the default
@@ -162,8 +223,8 @@ making such a change.
 * When the ``pythonX.X`` binaries are provided by a distribution, the
   ``python2`` and ``python3`` commands should refer to one of those files
   rather than being provided as a separate binary file.
-* It is strongly encouraged that distribution-specific packages use ``python2``
-  or ``python3`` rather than ``python``, even in code that is not intended to
+* It is strongly encouraged that distribution-specific packages use ``python3``
+  (or ``python2``) rather than ``python``, even in code that is not intended to
   operate on other distributions. This will reduce problems if the
   distribution later decides to change the version of the Python interpreter
   that the ``python`` command invokes, or if a sysadmin installs a custom
@@ -175,14 +236,13 @@ making such a change.
   versa. That way, if a sysadmin does decide to replace the installed
   ``python`` file, they can do so without inadvertently deleting the
   previously installed binary.
-* If the Python 2 interpreter becomes uncommon, scripts should nevertheless
+* When the Python 2 interpreter becomes uncommon, scripts should nevertheless
   continue to use the ``python3`` convention rather that just ``python``. This
   will ease transition in the event that yet another major version of Python
   is released.
 * If these conventions are adhered to, it will become the case that the
   ``python`` command is only executed in an interactive manner as a user
-  convenience, or to run scripts that are source compatible with both Python
-  2 and Python 3.
+  convenience.
 
 
 Backwards Compatibility
@@ -194,7 +254,7 @@ these commands. This is mostly a non-issue, since the sysadmin can simply
 create these symbolic links and avoid further problems. It is a significantly
 more obvious breakage than the sometimes cryptic errors that can arise when
 attempting to execute a script containing Python 2 specific syntax with a
-Python 3 interpreter.
+Python 3 interpreter or vice versa.
 
 
 Application to the CPython Reference Interpreter
@@ -209,7 +269,7 @@ items are relative symbolic links)::
     python -> python2 -> python2.7
     python-config -> python2-config -> python2.7-config
 
-Similar adjustments were made to the Mac OS X binary installer.
+Similar adjustments were made to the macOS binary installer.
 
 This feature first appeared in the default installation process in
 CPython 2.7.3.
@@ -252,8 +312,7 @@ Exclusion of MS Windows
 This PEP deliberately excludes any proposals relating to Microsoft Windows, as
 devising an equivalent solution for Windows was deemed too complex to handle
 here. PEP 397 and the related discussion on the python-dev mailing list
-address this issue (like this PEP, the PEP 397 launcher invokes Python 2 by
-default if versions of both Python 2 and 3 are installed on the system).
+address this issue.
 
 
 References
@@ -281,6 +340,13 @@ References
    minor edits
    (https://github.com/python/peps/pull/630)
 
+.. [8] Another update for PEP 394 -- The "python" Command on Unix-Like Systems
+   (https://mail.python.org/pipermail/python-dev/2019-February/156272.html)
+
+.. [9] The console_scripts Entry Point
+   (https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point)
+
+
 Copyright
 ===========
 This document has been placed in the public domain.