Merge branch 'main' into virtual-iterators

Author: markshannon

.github/workflows/mypy.yml

@@ -13,7 +13,13 @@ on:
       - "Lib/test/libregrtest/**"
       - "Lib/tomllib/**"
       - "Misc/mypy/**"
+      - "Tools/build/compute-changes.py"
+      - "Tools/build/deepfreeze.py"
       - "Tools/build/generate_sbom.py"
+      - "Tools/build/generate-build-details.py"
+      - "Tools/build/verify_ensurepip_wheels.py"
+      - "Tools/build/update_file.py"
+      - "Tools/build/umarshal.py"
       - "Tools/cases_generator/**"
       - "Tools/clinic/**"
       - "Tools/jit/**"

.github/workflows/reusable-context.yml

@@ -97,6 +97,7 @@ jobs:
       run: python Tools/build/compute-changes.py
       env:
         GITHUB_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+        GITHUB_EVENT_NAME: ${{ github.event_name }}
         CCF_TARGET_REF: ${{ github.base_ref || github.event.repository.default_branch }}
         CCF_HEAD_REF: ${{ github.event.pull_request.head.sha || github.sha }}
 

.pre-commit-config.yaml

@@ -47,6 +47,8 @@ repos:
         exclude: Lib/test/tokenizedata/coding20731.py
       - id: trailing-whitespace
         types_or: [c, inc, python, rst]
+      - id: trailing-whitespace
+        files: '\.(gram)$'
 
   - repo: https://github.com/python-jsonschema/check-jsonschema
     rev: 0.33.0

Doc/c-api/gcsupport.rst

@@ -180,9 +180,9 @@ provided.  In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse`
 must name its arguments exactly *visit* and *arg*:
 
 
-.. c:function:: void Py_VISIT(PyObject *o)
+.. c:macro:: Py_VISIT(o)
 
-   If *o* is not ``NULL``, call the *visit* callback, with arguments *o*
+   If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o*
    and *arg*.  If *visit* returns a non-zero value, then return it.
    Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
    look like::

Doc/c-api/import.rst

@@ -16,19 +16,6 @@ Importing Modules
    This is a wrapper around :c:func:`PyImport_Import()` which takes a
    :c:expr:`const char *` as an argument instead of a :c:expr:`PyObject *`.
 
-.. c:function:: PyObject* PyImport_ImportModuleNoBlock(const char *name)
-
-   This function is a deprecated alias of :c:func:`PyImport_ImportModule`.
-
-   .. versionchanged:: 3.3
-      This function used to fail immediately when the import lock was held
-      by another thread.  In Python 3.3 though, the locking scheme switched
-      to per-module locks for most purposes, so this function's special
-      behaviour isn't needed anymore.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyImport_ImportModule` instead.
-
 
 .. c:function:: PyObject* PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)
 

Doc/c-api/init.rst

@@ -77,10 +77,7 @@ The following functions can be safely called before Python is initialized:
 
    Despite their apparent similarity to some of the functions listed above,
    the following functions **should not be called** before the interpreter has
-   been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`,
-   :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`,
-   :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`,
-   :c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and
+   been initialized: :c:func:`Py_EncodeLocale`, :c:func:`PyEval_InitThreads`, and
    :c:func:`Py_RunMain`.
 
 
@@ -145,9 +142,6 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2.
    :c:member:`PyConfig.pathconfig_warnings` should be used instead, see
    :ref:`Python Initialization Configuration <init-config>`.
 
-   Suppress error messages when calculating the module search path in
-   :c:func:`Py_GetPath`.
-
    Private flag used by ``_freeze_module`` and ``frozenmain`` programs.
 
    .. deprecated-removed:: 3.12 3.15
@@ -203,7 +197,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2.
 
    Set by the :option:`-i` option.
 
-   .. deprecated:: 3.12
+   .. deprecated-removed:: 3.12 3.15
 
 .. c:var:: int Py_IsolatedFlag
 
@@ -586,7 +580,6 @@ Process-wide parameters
    .. index::
       single: Py_Initialize()
       single: main()
-      single: Py_GetPath()
 
    This API is kept for backward compatibility: setting
    :c:member:`PyConfig.program_name` should be used instead, see :ref:`Python
@@ -596,7 +589,7 @@ Process-wide parameters
    the first time, if it is called at all.  It tells the interpreter the value
    of the ``argv[0]`` argument to the :c:func:`main` function of the program
    (converted to wide characters).
-   This is used by :c:func:`Py_GetPath` and some other functions below to find
+   This is used by some other functions below to find
    the Python run-time libraries relative to the interpreter executable.  The
    default value is ``'python'``.  The argument should point to a
    zero-terminated wide character string in static storage whose contents will not
@@ -609,146 +602,6 @@ Process-wide parameters
    .. deprecated-removed:: 3.11 3.15
 
 
-.. c:function:: wchar_t* Py_GetProgramName()
-
-   Return the program name set with :c:member:`PyConfig.program_name`, or the default.
-   The returned string points into static storage; the caller should not modify its
-   value.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
-      (:data:`sys.executable`) instead.
-
-
-.. c:function:: wchar_t* Py_GetPrefix()
-
-   Return the *prefix* for installed platform-independent files. This is derived
-   through a number of complicated rules from the program name set with
-   :c:member:`PyConfig.program_name` and some environment variables; for example, if the
-   program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
-   returned string points into static storage; the caller should not modify its
-   value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
-   script at build time.  The value is available to Python code as ``sys.base_prefix``.
-   It is only useful on Unix.  See also the next function.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>`
-      (:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix")
-      <PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments
-      <venv-def>` need to be handled.
-
-
-.. c:function:: wchar_t* Py_GetExecPrefix()
-
-   Return the *exec-prefix* for installed platform-*dependent* files.  This is
-   derived through a number of complicated rules from the program name set with
-   :c:member:`PyConfig.program_name` and some environment variables; for example, if the
-   program name is ``'/usr/local/bin/python'``, the exec-prefix is
-   ``'/usr/local'``.  The returned string points into static storage; the caller
-   should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
-   argument to the :program:`configure` script at build  time.  The value is
-   available to Python code as ``sys.base_exec_prefix``.  It is only useful on
-   Unix.
-
-   Background: The exec-prefix differs from the prefix when platform dependent
-   files (such as executables and shared libraries) are installed in a different
-   directory tree.  In a typical installation, platform dependent files may be
-   installed in the :file:`/usr/local/plat` subtree while platform independent may
-   be installed in :file:`/usr/local`.
-
-   Generally speaking, a platform is a combination of hardware and software
-   families, e.g.  Sparc machines running the Solaris 2.x operating system are
-   considered the same platform, but Intel machines running Solaris 2.x are another
-   platform, and Intel machines running Linux are yet another platform.  Different
-   major revisions of the same operating system generally also form different
-   platforms.  Non-Unix operating systems are a different story; the installation
-   strategies on those systems are so different that the prefix and exec-prefix are
-   meaningless, and set to the empty string. Note that compiled Python bytecode
-   files are platform independent (but not independent from the Python version by
-   which they were compiled!).
-
-   System administrators will know how to configure the :program:`mount` or
-   :program:`automount` programs to share :file:`/usr/local` between platforms
-   while having :file:`/usr/local/plat` be a different filesystem for each
-   platform.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>`
-      (:data:`sys.base_exec_prefix`) instead. Use
-      :c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>`
-      (:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need
-      to be handled.
-
-.. c:function:: wchar_t* Py_GetProgramFullPath()
-
-   .. index::
-      single: executable (in module sys)
-
-   Return the full program name of the Python executable; this is  computed as a
-   side-effect of deriving the default module search path  from the program name
-   (set by :c:member:`PyConfig.program_name`). The returned string points into
-   static storage; the caller should not modify its value.  The value is available
-   to Python code as ``sys.executable``.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
-      (:data:`sys.executable`) instead.
-
-
-.. c:function:: wchar_t* Py_GetPath()
-
-   .. index::
-      triple: module; search; path
-      single: path (in module sys)
-
-   Return the default module search path; this is computed from the program name
-   (set by :c:member:`PyConfig.program_name`) and some environment variables.
-   The returned string consists of a series of directory names separated by a
-   platform dependent delimiter character.  The delimiter character is ``':'``
-   on Unix and macOS, ``';'`` on Windows.  The returned string points into
-   static storage; the caller should not modify its value.  The list
-   :data:`sys.path` is initialized with this value on interpreter startup; it
-   can be (and usually is) modified later to change the search path for loading
-   modules.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. XXX should give the exact rules
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>`
-      (:data:`sys.path`) instead.
-
 .. c:function:: const char* Py_GetVersion()
 
    Return the version of this Python interpreter.  This is a string that looks
@@ -919,23 +772,6 @@ Process-wide parameters
    .. deprecated-removed:: 3.11 3.15
 
 
-.. c:function:: wchar_t* Py_GetPythonHome()
-
-   Return the default "home", that is, the value set by
-   :c:member:`PyConfig.home`, or the value of the :envvar:`PYTHONHOME`
-   environment variable if it is set.
-
-   This function should not be called before :c:func:`Py_Initialize`, otherwise
-   it returns ``NULL``.
-
-   .. versionchanged:: 3.10
-      It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the
-      :envvar:`PYTHONHOME` environment variable instead.
-
-
 .. _threads:
 
 Thread State and the Global Interpreter Lock
@@ -1083,8 +919,36 @@ Note that the ``PyGILState_*`` functions assume there is only one global
 interpreter (created automatically by :c:func:`Py_Initialize`).  Python
 supports the creation of additional interpreters (using
 :c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the
-``PyGILState_*`` API is unsupported.
+``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure`
+and similar functions default to :term:`attaching <attached thread state>` a
+:term:`thread state` for the main interpreter, meaning that the thread can't safely
+interact with the calling subinterpreter.
+
+Supporting subinterpreters in non-Python threads
+------------------------------------------------
+
+If you would like to support subinterpreters with non-Python created threads, you
+must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*``
+API.
+
+In particular, you must store the interpreter state from the calling
+function and pass it to :c:func:`PyThreadState_New`, which will ensure that
+the :term:`thread state` is targeting the correct interpreter::
+
+   /* The return value of PyInterpreterState_Get() from the
+      function that created this thread. */
+   PyInterpreterState *interp = ThreadData->interp;
+   PyThreadState *tstate = PyThreadState_New(interp);
+   PyThreadState_Swap(tstate);
+
+   /* GIL of the subinterpreter is now held.
+      Perform Python actions here. */
+   result = CallSomeFunction();
+   /* evaluate result or handle exception */
 
+   /* Destroy the thread state. No Python API allowed beyond this point. */
+   PyThreadState_Clear(tstate);
+   PyThreadState_DeleteCurrent();
 
 .. _fork-and-threads:
 
@@ -1261,6 +1125,10 @@ code, or when embedding the Python interpreter:
    .. seealso:
       :c:func:`PyEval_ReleaseThread`
 
+   .. note::
+      Similar to :c:func:`PyGILState_Ensure`, this function will hang the
+      thread if the runtime is finalizing.
+
 
 The following functions use thread-local storage, and are not compatible
 with sub-interpreters:
@@ -1287,10 +1155,10 @@ with sub-interpreters:
    When the function returns, there will be an :term:`attached thread state`
    and the thread will be able to call arbitrary Python code.  Failure is a fatal error.
 
-   .. note::
-      Calling this function from a thread when the runtime is finalizing will
-      hang the thread until the program exits, even if the thread was not
-      created by Python.  Refer to
+   .. warning::
+      Calling this function when the runtime is finalizing is unsafe. Doing
+      so will either hang the thread until the program ends, or fully crash
+      the interpreter in rare cases. Refer to
       :ref:`cautions-regarding-runtime-finalization` for more details.
 
    .. versionchanged:: 3.14
@@ -1307,28 +1175,37 @@ with sub-interpreters:
    Every call to :c:func:`PyGILState_Ensure` must be matched by a call to
    :c:func:`PyGILState_Release` on the same thread.
 
-
 .. c:function:: PyThreadState* PyGILState_GetThisThreadState()
 
    Get the :term:`attached thread state` for this thread.  May return ``NULL`` if no
    GILState API has been used on the current thread.  Note that the main thread
    always has such a thread-state, even if no auto-thread-state call has been
    made on the main thread.  This is mainly a helper/diagnostic function.
 
-   .. seealso: :c:func:`PyThreadState_Get``
+   .. note::
+      This function does not account for :term:`thread states <thread state>` created
+      by something other than :c:func:`PyGILState_Ensure` (such as :c:func:`PyThreadState_New`).
+      Prefer :c:func:`PyThreadState_Get` or :c:func:`PyThreadState_GetUnchecked`
+      for most cases.
 
+   .. seealso: :c:func:`PyThreadState_Get``
 
 .. c:function:: int PyGILState_Check()
 
    Return ``1`` if the current thread is holding the :term:`GIL` and ``0`` otherwise.
    This function can be called from any thread at any time.
-   Only if it has had its Python thread state initialized and currently is
-   holding the :term:`GIL` will it return ``1``.
+   Only if it has had its :term:`thread state <attached thread state>` initialized
+   via :c:func:`PyGILState_Ensure` will it return ``1``.
    This is mainly a helper/diagnostic function.  It can be useful
    for example in callback contexts or memory allocation functions when
    knowing that the :term:`GIL` is locked can allow the caller to perform sensitive
    actions or otherwise behave differently.
 
+   .. note::
+      If the current Python process has ever created a subinterpreter, this
+      function will *always* return ``1``. Prefer :c:func:`PyThreadState_GetUnchecked`
+      for most cases.
+
    .. versionadded:: 3.4