PEP 394: Allow for more flexibility in handling /usr/bin/python

pep-0394.txt

@@ -5,104 +5,181 @@ Last-Modified: $Date$
 Author: Kerrick Staley <[email protected]>,
         Nick Coghlan <[email protected]>,
         Barry Warsaw <[email protected]>,
-        Petr Viktorin <[email protected]>
+        Petr Viktorin <[email protected]>,
+        Miro Hrončok <[email protected]>,
 Status: Active
 Type: Informational
 Content-Type: text/x-rst
 Created: 02-Mar-2011
-Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018
+Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018,
+              12-Apr-2019
 Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html
 
 
 Abstract
 ========
 
 This PEP provides a convention to ensure that Python scripts can continue to
-be portable across ``*nix`` systems, regardless of the default version of the
-Python interpreter (i.e. the version invoked by the ``python`` command).
+be portable across ``*nix`` systems, regardless of the version of the Python
+interpreter invoked by the ``python`` command.
 
 * ``python2`` will refer to some version of Python 2.x.
 * ``python3`` will refer to some version of Python 3.x.
-* for the time being, all distributions *should* ensure that ``python``,
-  if installed, refers to the same target as ``python2``, unless the user
-  deliberately overrides this or a virtual environment is active.
-* however, end users should be aware that ``python`` refers to ``python3``
-  on at least Arch Linux (that change is what prompted the creation of this
-  PEP), so ``python`` should be used in the shebang line only for scripts
-  that are source compatible with both Python 2 and 3.
-* in preparation for an eventual change in the default version of Python,
-  Python 2 only scripts should either be updated to be source compatible
-  with Python 3 or else to use ``python2`` in the shebang line.
-
+* Distributions can choose whether ``python`` is installed and, if it is,
+  whether it refers to the same target as ``python2`` or ``python3``.
+* End users should be aware that the ``python`` command is inconsistent
+  across Unix-like systems.
+* In activated virtual environments, ``python`` always refers to the
+  environment's Python version.
+* Cross-platform scripts should prefer the ``python3`` shebang even if they
+  support both Python 3 and Python 2.
+* Cross-platform scripts only supporting Python 2 should use the ``python2``
+  shebang.
 
 Recommendation
 ==============
 
-* Unix-like software distributions (including systems like Mac OS X and
+For distributors
+----------------
+
+* Unix-like software distributions (including systems like macOS and
   Cygwin) should install the ``python2`` command into the default path
   whenever a version of the Python 2 interpreter is installed, and the same
   for ``python3`` and the Python 3 interpreter.
 * When  invoked, ``python2`` should run some version of the Python 2
   interpreter, and ``python3`` should run some version of the Python 3
   interpreter.
-* If the ``python`` command is installed, it should invoke the same version of
-  Python as the ``python2`` command (however, note that some distributions
-  have already chosen to have ``python`` implement the ``python3``
-  command; see the `Rationale`_ and `Migration Notes`_ below).
-* The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should
-  likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``,
-  with the original commands invoking these versions by default, but possibly
-  invoking the Python 3.x versions instead if configured to do so by the
+* If the ``python`` command is installed, it should either invoke the same
+  version of Python as the ``python3`` command or as the ``python2``
+  command.
+* Distributors may choose to set the behavior of the ``python`` command,
+  not provide it at all, or make it configurable by the user or
   system administrator.
-* In order to tolerate differences across platforms, all new code that needs
-  to invoke the Python interpreter should not specify ``python``, but rather
-  should specify either ``python2`` or ``python3`` (or the more specific
-  ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_).
-  This distinction should be made in shebangs, when invoking from a shell
-  script, when invoking via the system() call, or when invoking in any other
-  context.
-* One exception to this is scripts that are deliberately written to be source
-  compatible with both Python 2.x and 3.x. Such scripts may continue to use
-  ``python`` on their shebang line.
-* When packaging software that is source compatible with both versions,
-  distributions may change such ``python`` shebangs to ``python3``.
-  This ensures software is used with the latest version of
-  Python available, and it can remove a dependency on Python 2.
+* The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should
+  likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``;
+  Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``.
+  The original commands should either invoke the same version of Python
+  as the ``python`` command, or not be available at all.
+* When packaging software, distributors are encouraged to change less specific
+  shebangs to more specific ones. This ensures software is used with the latest
+  version of Python available, and it can remove a dependency on Python 2.
+  The specifics are left on the distributors.
+  Examples of this include changing ``python`` shebangs to ``python3``
+  when Python 3.x is supported, changing ``python`` shebangs to ``python2``
+  when Python 3.x is not yet supported, or changing ``python3`` shebangs to
+  ``python3.8`` if the software is built with Python 3.8.
+* When a virtual environment (created by the PEP 405 ``venv`` package or a
+  similar tool such as ``virtualenv`` or ``conda``) is active, the ``python``
+  command should refer to the virtual environment's interpreter and should
+  always be available.
+  The ``python3`` or ``python2`` command (according to the environment's
+  interpreter) should also be available.
+
+For developers
+--------------
+
 * When reinvoking the interpreter from a Python script, querying
   ``sys.executable`` to avoid hardcoded assumptions regarding the
   interpreter location remains the preferred approach.
-* In controlled environments aimed at expert users, where being explicit
-  is valued over user experience (for example, in test environments and
-  package build systems), distributions may choose to not provide the
-  ``python`` command even if ``python2`` is available.
-  (All software in such a controlled environment must use ``python3`` or
-  ``python2`` rather than ``python``, which means scripts that deliberately
-  use ``python`` need to be modified for such environments.)
-* When a virtual environment (created by the PEP 405 ``venv`` package or a
-  similar tool) is active, the ``python`` command should refer to the
-  virtual environment's interpreter. In other words, activating a virtual
-  environment counts as deliberate user action to change the default
-  ``python`` interpreter.
+* While far from being universally available, ``python`` remains the
+  preferred spelling for explicitly invoking Python, as this is the
+  spelling that virtual environments make consistently available
+  across different platforms and Python installations.
+* In shebang lines, the preferred spelling is ``/usr/bin/env python``,
+  as this instructs the script to respect the active virtual environment.
+* For Python 3 only scripts that do not support being executed on Python
+  2 at all, it is recommended to instead use ``python3`` and
+  ``/usr/bin/env python3``, as these will never invoke Python 2, and are
+  expected to work for both Python 3 virtual environments and Python 3
+  system installations
+* For Python 2 only scripts that do not support being executed on Python
+  3 at all, it is recommended to instead use ``python2`` and
+  ``/usr/bin/env python2``, as these will never invoke Python 3, and are
+  expected to work for Python 2 virtual environments and at least some
+  Python 2 system installations.
+* In cases where the script is expected to be executed outside virtual
+  environments, developers will need to be aware of the following
+  discrepancies across platforms and installation methods:
+
+  * Older Linux distributions will provide a ``python`` command that
+    refers to Python 2. Most of these distros do *not* provide a
+    ``python2`` command.
+  * Some newer Linux distributions will provide a ``python`` command that
+    refers to Python 3
+  * Some Linux distributions will not provide a ``python`` command at
+    all by default, but will provide a ``python3`` command by default
+
+* When potentially targeting these environments, developers may either
+  use a Python package installation tool that rewrites shebang lines for
+  the installed environment, provide instructions on updating shebang lines
+  interactively, or else use more specific shebang lines that are
+  tailored to the target environment.
+* Scripts targeting both “*old systems*” and systems without the default
+  ``python`` command need to make a compromise and document this situation.
+  Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar
+  means) is the recommended workaround for this problem.
+* Applications designed exclusively for a specific environment (such as
+  a container or virtual environment) may continue to use the ``python``
+  command name.
 
 These recommendations are the outcome of the relevant python-dev discussions
 in March and July 2011 ([1]_, [2]_), February 2012 ([4]_),
-September 2014 ([6]_), and discussion on GitHub in April 2018 ([7]_).
+September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_),
+on python-dev in February 2019 ([8]_), and during the PEP update review
+in May 2019 ([10]_).
 
 
-Rationale
-=========
+History of this PEP
+===================
 
-This recommendation is needed as, even though the majority of distributions
-still alias the ``python`` command to Python 2, some now alias it to
+In 2011, the majority of distributions
+aliased the ``python`` command to Python 2, but some started switching it to
 Python 3 ([5]_). As some of the former distributions did not provide a
 ``python2`` command by default, there was previously no way for Python 2 code
 (or any code that invokes the Python 2 interpreter directly rather than via
 ``sys.executable``) to reliably run on all Unix-like systems without
 modification, as the ``python`` command would invoke the wrong interpreter
 version on some systems, and the ``python2`` command would fail completely
-on others. The recommendations in this PEP provide a very simple mechanism
+on others. This PEP originally provided a very simple mechanism
 to restore cross-platform support, with minimal additional work required
-on the part of distribution maintainers.
+on the part of distribution maintainers. Simplified, the recommendation was:
+
+1. The ``python`` command was preferred for code compatible with both
+   Python 2 and 3 (since it was available on all systems, even those that
+   already aliased it to Python 3).
+2. The ``python`` command should always invoke Python 2 (to prevent
+   hard-to-diagnose errors when Python 2 code is run on Python 3).
+3. The ``python2`` and ``python3`` commands should be available to specify
+   the version explicitly.
+
+However, these recommendations implicitly assumed that Python 2 would always be
+available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404),
+distributions are making Python 2 optional or removing it entirely.
+This means either removing the ``python`` command or switching it to invoke
+Python 3. Some distributors also decided that their users were better served by
+ignoring the PEP's recommendations, and provided system administrators with the
+freedom to configure their systems based on the needs of their particular
+environment.
+
+
+.. _rationale:
+
+Current Rationale
+=================
+
+As of 2019, activating a Python virtual environment (or its functional
+equivalent) is the best way to obtain a consistent cross-platform and
+cross-distribution experience.
+
+Accordingly, it is entirely reasonable for publishers to expect the
+availability of such an environment to get their software working
+correctly, and push responsibility for handling other environments
+(such as system Python installations) onto consumers of the software.
+
+As part of that however, it is also appropriate to give Python
+distributors the flexibility they need in order to make the
+behaviour of their systems as similar as possible to the behaviour
+of an activated virtual environment.
 
 
 Future Changes to this Recommendation
@@ -111,7 +188,7 @@ Future Changes to this Recommendation
 This recommendation will be periodically reviewed over the next few years,
 and updated when the core development team judges it appropriate. As a
 point of reference, regular maintenance releases for the Python 2.7 series
-will continue until at least 2020.
+will continue until January 2020.
 
 
 Migration Notes
@@ -129,27 +206,25 @@ making such a change.
   and other users. Updating the ``python`` command to invoke ``python3``
   by default indicates that a distribution is willing to break such scripts
   with errors that are potentially quite confusing for users that aren't
-  yet familiar with the backwards incompatible changes in Python 3. For
+  familiar with the backwards incompatible changes in Python 3. For
   example, while the change of ``print`` from a statement to a builtin
   function is relatively simple for automated converters to handle, the
-  SyntaxError from attempting to use the Python 2 notation in versions of
-  Python 3 prior to 3.4.2 is thoroughly confusing if you aren't already
-  aware of the change::
+  SyntaxError from attempting to use the Python 2 notation in Python 3
+  may be confusing for users that are not aware of the change::
 
       $ python3 -c 'print "Hello, world!"'
         File "<string>", line 1
           print "Hello, world!"
-                              ^
-      SyntaxError: invalid syntax
-
-  (In Python 3.4.2+, that generic error message has been replaced with the
-  more explicit "SyntaxError: Missing parentheses in call to 'print'")
-* Avoiding breakage of such third party scripts is the key reason this
-  PEP recommends that ``python`` continue to refer to ``python2`` for the
-  time being. Until the conventions described in this PEP are more widely
-  adopted, having ``python`` invoke ``python2`` will remain the recommended
-  option.
-* The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on
+                ^
+      SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello, world!")?
+
+  While this might be obvious for experienced Pythonistas, such scripts
+  might even be run by people who are not familiar with Python at all.
+  Avoiding breakage of such third party scripts was the key reason this
+  PEP used to recommend that ``python`` continue to refer to ``python2``.
+* The error message ``python: command not found`` tends to be surprisingly
+  actionable, even for people unfamiliar with Python.
+* The ``pythonX.X`` (e.g. ``python3.6``) commands exist on modern systems, on
   which they invoke specific minor versions of the Python interpreter. It
   can be useful for distribution-specific packages to take advantage of these
   utilities if they exist, since it will prevent code breakage if the default
@@ -162,8 +237,8 @@ making such a change.
 * When the ``pythonX.X`` binaries are provided by a distribution, the
   ``python2`` and ``python3`` commands should refer to one of those files
   rather than being provided as a separate binary file.
-* It is strongly encouraged that distribution-specific packages use ``python2``
-  or ``python3`` rather than ``python``, even in code that is not intended to
+* It is strongly encouraged that distribution-specific packages use ``python3``
+  (or ``python2``) rather than ``python``, even in code that is not intended to
   operate on other distributions. This will reduce problems if the
   distribution later decides to change the version of the Python interpreter
   that the ``python`` command invokes, or if a sysadmin installs a custom
@@ -175,14 +250,17 @@ making such a change.
   versa. That way, if a sysadmin does decide to replace the installed
   ``python`` file, they can do so without inadvertently deleting the
   previously installed binary.
-* If the Python 2 interpreter becomes uncommon, scripts should nevertheless
-  continue to use the ``python3`` convention rather that just ``python``. This
-  will ease transition in the event that yet another major version of Python
-  is released.
+* Even as the Python 2 interpreter becomes less common, scripts should
+  nevertheless continue to use the ``python3`` convention rather than just
+  ``python``. In the event of a future Python 4.0 release, distributors may
+  then choose to simply alias ``python3`` to the same binary as ``python4``
+  (if the degree of change will be the same as in any other Python feature
+  release), or else they may choose to maintain parallel ``python3`` and
+  ``python4`` stacks for a time (depending on the needs of their specific
+  audience).
 * If these conventions are adhered to, it will become the case that the
   ``python`` command is only executed in an interactive manner as a user
-  convenience, or to run scripts that are source compatible with both Python
-  2 and Python 3.
+  convenience, or else when using a virtual environment or similar mechanism.
 
 
 Backwards Compatibility
@@ -194,7 +272,7 @@ these commands. This is mostly a non-issue, since the sysadmin can simply
 create these symbolic links and avoid further problems. It is a significantly
 more obvious breakage than the sometimes cryptic errors that can arise when
 attempting to execute a script containing Python 2 specific syntax with a
-Python 3 interpreter.
+Python 3 interpreter or vice versa.
 
 
 Application to the CPython Reference Interpreter
@@ -209,7 +287,7 @@ items are relative symbolic links)::
     python -> python2 -> python2.7
     python-config -> python2-config -> python2.7-config
 
-Similar adjustments were made to the Mac OS X binary installer.
+Similar adjustments were made to the macOS binary installer.
 
 This feature first appeared in the default installation process in
 CPython 2.7.3.
@@ -252,8 +330,7 @@ Exclusion of MS Windows
 This PEP deliberately excludes any proposals relating to Microsoft Windows, as
 devising an equivalent solution for Windows was deemed too complex to handle
 here. PEP 397 and the related discussion on the python-dev mailing list
-address this issue (like this PEP, the PEP 397 launcher invokes Python 2 by
-default if versions of both Python 2 and 3 are installed on the system).
+address this issue.
 
 
 References
@@ -281,6 +358,16 @@ References
    minor edits
    (https://github.com/python/peps/pull/630)
 
+.. [8] Another update for PEP 394 -- The "python" Command on Unix-Like Systems
+   (https://mail.python.org/pipermail/python-dev/2019-February/156272.html)
+
+.. [9] The console_scripts Entry Point
+   (https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point)
+
+.. [10] May 2019 PEP update review
+   (https://github.com/python/peps/pull/989)
+
+
 Copyright
 ===========
 This document has been placed in the public domain.