Merge branch 'main' into typewatch

* main:
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)

Author: carljm

.github/CODEOWNERS

@@ -4,6 +4,9 @@
 # It uses the same pattern rule for gitignore file
 # https://git-scm.com/docs/gitignore#_pattern_format
 
+# GitHub
+.github/**                    @ezio-melotti
+
 # asyncio
 **/*asyncio*                  @1st1 @asvetlov @gvanrossum
 

.github/workflows/project-updater.yml

@@ -6,6 +6,9 @@ on:
       - opened
       - labeled
 
+permissions:
+  contents: read
+
 jobs:
   add-to-project:
     name: Add issues to projects

Doc/c-api/dict.rst

@@ -246,24 +246,41 @@ Dictionary Objects
    of error (e.g. no more watcher IDs available), return ``-1`` and set an
    exception.
 
+   .. versionadded:: 3.12
+
 .. c:function:: int PyDict_ClearWatcher(int watcher_id)
 
    Clear watcher identified by *watcher_id* previously returned from
    :c:func:`PyDict_AddWatcher`. Return ``0`` on success, ``-1`` on error (e.g.
    if the given *watcher_id* was never registered.)
 
+   .. versionadded:: 3.12
+
 .. c:function:: int PyDict_Watch(int watcher_id, PyObject *dict)
 
    Mark dictionary *dict* as watched. The callback granted *watcher_id* by
    :c:func:`PyDict_AddWatcher` will be called when *dict* is modified or
-   deallocated.
+   deallocated. Return ``0`` on success or ``-1`` on error.
+
+   .. versionadded:: 3.12
+
+.. c:function:: int PyDict_Unwatch(int watcher_id, PyObject *dict)
+
+   Mark dictionary *dict* as no longer watched. The callback granted
+   *watcher_id* by :c:func:`PyDict_AddWatcher` will no longer be called when
+   *dict* is modified or deallocated. The dict must previously have been
+   watched by this watcher. Return ``0`` on success or ``-1`` on error.
+
+   .. versionadded:: 3.12
 
 .. c:type:: PyDict_WatchEvent
 
    Enumeration of possible dictionary watcher events: ``PyDict_EVENT_ADDED``,
    ``PyDict_EVENT_MODIFIED``, ``PyDict_EVENT_DELETED``, ``PyDict_EVENT_CLONED``,
    ``PyDict_EVENT_CLEARED``, or ``PyDict_EVENT_DEALLOCATED``.
 
+   .. versionadded:: 3.12
+
 .. c:type:: int (*PyDict_WatchCallback)(PyDict_WatchEvent event, PyObject *dict, PyObject *key, PyObject *new_value)
 
    Type of a dict watcher callback function.
@@ -289,3 +306,5 @@ Dictionary Objects
    If the callback returns with an exception set, it must return ``-1``; this
    exception will be printed as an unraisable exception using
    :c:func:`PyErr_WriteUnraisable`. Otherwise it should return ``0``.
+
+   .. versionadded:: 3.12

Doc/library/http.server.rst

@@ -392,8 +392,8 @@ provides three different variants:
       contents of the file are output. If the file's MIME type starts with
       ``text/`` the file is opened in text mode; otherwise binary mode is used.
 
-      For example usage, see the implementation of the :func:`test` function
-      invocation in the :mod:`http.server` module.
+      For example usage, see the implementation of the ``test`` function
+      in :source:`Lib/http/server.py`.
 
       .. versionchanged:: 3.7
          Support of the ``'If-Modified-Since'`` header.

Doc/license.rst

@@ -302,7 +302,8 @@ for third-party software incorporated in the Python distribution.
 Mersenne Twister
 ----------------
 
-The :mod:`_random` module includes code based on a download from
+The :mod:`!_random` C extension underlying the :mod:`random` module
+includes code based on a download from
 http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html. The following are
 the verbatim comments from the original code::
 
@@ -819,7 +820,8 @@ sources unless the build is configured ``--with-system-expat``::
 libffi
 ------
 
-The :mod:`_ctypes` extension is built using an included copy of the libffi
+The :mod:`!_ctypes` C extension underlying the :mod:`ctypes` module
+is built using an included copy of the libffi
 sources unless the build is configured ``--with-system-libffi``::
 
    Copyright (c) 1996-2008  Red Hat, Inc and others.
@@ -920,7 +922,8 @@ on the cfuhash project::
 libmpdec
 --------
 
-The :mod:`_decimal` module is built using an included copy of the libmpdec
+The :mod:`!_decimal` C extension underlying the :mod:`decimal` module
+is built using an included copy of the libmpdec
 library unless the build is configured ``--with-system-libmpdec``::
 
    Copyright (c) 2008-2020 Stefan Krah. All rights reserved.

Doc/reference/expressions.rst

@@ -1741,6 +1741,12 @@ returns a boolean value regardless of the type of its argument
 (for example, ``not 'foo'`` produces ``False`` rather than ``''``.)
 
 
+.. index::
+   single: := (colon equals)
+   single: assignment expression
+   single: walrus operator
+   single: named expression
+
 Assignment expressions
 ======================
 

Doc/whatsnew/3.12.rst

@@ -93,6 +93,13 @@ Other Language Changes
   when parsing source code containing null bytes. (Contributed by Pablo Galindo
   in :gh:`96670`.)
 
+* The Garbage Collector now runs only on the eval breaker mechanism of the
+  Python bytecode evaluation loop instead on object allocations. The GC can
+  also run when :c:func:`PyErr_CheckSignals` is called so C extensions that
+  need to run for a long time without executing any Python code also have a
+  chance to execute the GC periodically. (Contributed by Pablo Galindo in
+  :gh:`97922`.)
+
 New Modules
 ===========
 
@@ -546,6 +553,11 @@ New Features
   which sets the vectorcall field of a given :c:type:`PyFunctionObject`.
   (Contributed by Andrew Frost in :gh:`92257`.)
 
+* The C API now permits registering callbacks via :c:func:`PyDict_AddWatcher`,
+  :c:func:`PyDict_AddWatch` and related APIs to be called whenever a dictionary
+  is modified. This is intended for use by optimizing interpreters, JIT
+  compilers, or debuggers.
+
 Porting to Python 3.12
 ----------------------
 

Include/cpython/dictobject.h

@@ -106,3 +106,4 @@ PyAPI_FUNC(int) PyDict_ClearWatcher(int watcher_id);
 
 // Mark given dictionary as "watched" (callback will be called if it is modified)
 PyAPI_FUNC(int) PyDict_Watch(int watcher_id, PyObject* dict);
+PyAPI_FUNC(int) PyDict_Unwatch(int watcher_id, PyObject* dict);

Include/internal/pycore_gc.h

@@ -202,6 +202,8 @@ extern void _PyList_ClearFreeList(PyInterpreterState *interp);
 extern void _PyDict_ClearFreeList(PyInterpreterState *interp);
 extern void _PyAsyncGen_ClearFreeLists(PyInterpreterState *interp);
 extern void _PyContext_ClearFreeList(PyInterpreterState *interp);
+extern void _Py_ScheduleGC(PyInterpreterState *interp);
+extern void _Py_RunGC(PyThreadState *tstate);
 
 #ifdef __cplusplus
 }

Include/internal/pycore_interp.h

@@ -49,6 +49,8 @@ struct _ceval_state {
     _Py_atomic_int eval_breaker;
     /* Request for dropping the GIL */
     _Py_atomic_int gil_drop_request;
+    /* The GC is ready to be executed */
+    _Py_atomic_int gc_scheduled;
     struct _pending_calls pending;
 };
 

Lib/asyncio/unix_events.py

@@ -912,46 +912,29 @@ class PidfdChildWatcher(AbstractChildWatcher):
     recent (5.3+) kernels.
     """
 
-    def __init__(self):
-        self._loop = None
-        self._callbacks = {}
-
     def __enter__(self):
         return self
 
     def __exit__(self, exc_type, exc_value, exc_traceback):
         pass
 
     def is_active(self):
-        return self._loop is not None and self._loop.is_running()
+        return True
 
     def close(self):
-        self.attach_loop(None)
+        pass
 
     def attach_loop(self, loop):
-        if self._loop is not None and loop is None and self._callbacks:
-            warnings.warn(
-                'A loop is being detached '
-                'from a child watcher with pending handlers',
-                RuntimeWarning)
-        for pidfd, _, _ in self._callbacks.values():
-            self._loop._remove_reader(pidfd)
-            os.close(pidfd)
-        self._callbacks.clear()
-        self._loop = loop
+        pass
 
     def add_child_handler(self, pid, callback, *args):
-        existing = self._callbacks.get(pid)
-        if existing is not None:
-            self._callbacks[pid] = existing[0], callback, args
-        else:
-            pidfd = os.pidfd_open(pid)
-            self._loop._add_reader(pidfd, self._do_wait, pid)
-            self._callbacks[pid] = pidfd, callback, args
+        loop = events.get_running_loop()
+        pidfd = os.pidfd_open(pid)
+        loop._add_reader(pidfd, self._do_wait, pid, pidfd, callback, args)
 
-    def _do_wait(self, pid):
-        pidfd, callback, args = self._callbacks.pop(pid)
-        self._loop._remove_reader(pidfd)
+    def _do_wait(self, pid, pidfd, callback, args):
+        loop = events.get_running_loop()
+        loop._remove_reader(pidfd)
         try:
             _, status = os.waitpid(pid, 0)
         except ChildProcessError:
@@ -969,12 +952,9 @@ def _do_wait(self, pid):
         callback(pid, returncode, *args)
 
     def remove_child_handler(self, pid):
-        try:
-            pidfd, _, _ = self._callbacks.pop(pid)
-        except KeyError:
-            return False
-        self._loop._remove_reader(pidfd)
-        os.close(pidfd)
+        # asyncio never calls remove_child_handler() !!!
+        # The method is no-op but is implemented because
+        # abstract base classes require it.
         return True
 
 
@@ -1423,6 +1403,17 @@ def _do_waitpid(self, loop, expected_pid, callback, args):
 
         self._threads.pop(expected_pid)
 
+def can_use_pidfd():
+    if not hasattr(os, 'pidfd_open'):
+        return False
+    try:
+        pid = os.getpid()
+        os.close(os.pidfd_open(pid, 0))
+    except OSError:
+        # blocked by security policy like SECCOMP
+        return False
+    return True
+
 
 class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy):
     """UNIX event loop policy with a watcher for child processes."""
@@ -1435,7 +1426,10 @@ def __init__(self):
     def _init_watcher(self):
         with events._lock:
             if self._watcher is None:  # pragma: no branch
-                self._watcher = ThreadedChildWatcher()
+                if can_use_pidfd():
+                    self._watcher = PidfdChildWatcher()
+                else:
+                    self._watcher = ThreadedChildWatcher()
                 if threading.current_thread() is threading.main_thread():
                     self._watcher.attach_loop(self._local._loop)
 

Lib/test/test_asyncio/test_subprocess.py

@@ -4,6 +4,7 @@
 import sys
 import unittest
 import warnings
+import functools
 from unittest import mock
 
 import asyncio
@@ -30,6 +31,19 @@
               'sys.stdout.buffer.write(data)'))]
 
 
+@functools.cache
+def _has_pidfd_support():
+    if not hasattr(os, 'pidfd_open'):
+        return False
+
+    try:
+        os.close(os.pidfd_open(os.getpid()))
+    except OSError:
+        return False
+
+    return True
+
+
 def tearDownModule():
     asyncio.set_event_loop_policy(None)
 
@@ -708,17 +722,8 @@ class SubprocessFastWatcherTests(SubprocessWatcherMixin,
 
         Watcher = unix_events.FastChildWatcher
 
-    def has_pidfd_support():
-        if not hasattr(os, 'pidfd_open'):
-            return False
-        try:
-            os.close(os.pidfd_open(os.getpid()))
-        except OSError:
-            return False
-        return True
-
     @unittest.skipUnless(
-        has_pidfd_support(),
+        _has_pidfd_support(),
         "operating system does not support pidfds",
     )
     class SubprocessPidfdWatcherTests(SubprocessWatcherMixin,
@@ -751,6 +756,35 @@ async def execute():
                 mock.call.__exit__(RuntimeError, mock.ANY, mock.ANY),
             ])
 
+
+        @unittest.skipUnless(
+            _has_pidfd_support(),
+            "operating system does not support pidfds",
+        )
+        def test_create_subprocess_with_pidfd(self):
+            async def in_thread():
+                proc = await asyncio.create_subprocess_exec(
+                    *PROGRAM_CAT,
+                    stdin=subprocess.PIPE,
+                    stdout=subprocess.PIPE,
+                )
+                stdout, stderr = await proc.communicate(b"some data")
+                return proc.returncode, stdout
+
+            async def main():
+                # asyncio.Runner did not call asyncio.set_event_loop()
+                with self.assertRaises(RuntimeError):
+                    asyncio.get_event_loop_policy().get_event_loop()
+                return await asyncio.to_thread(asyncio.run, in_thread())
+
+            asyncio.set_child_watcher(asyncio.PidfdChildWatcher())
+            try:
+                with asyncio.Runner(loop_factory=asyncio.new_event_loop) as runner:
+                    returncode, stdout = runner.run(main())
+                self.assertEqual(returncode, 0)
+                self.assertEqual(stdout, b'some data')
+            finally:
+                asyncio.set_child_watcher(None)
 else:
     # Windows
     class SubprocessProactorTests(SubprocessMixin, test_utils.TestCase):

Lib/test/test_asyncio/test_unix_events.py

@@ -1702,14 +1702,26 @@ def create_policy(self):
     def test_get_default_child_watcher(self):
         policy = self.create_policy()
         self.assertIsNone(policy._watcher)
-
+        unix_events.can_use_pidfd = mock.Mock()
+        unix_events.can_use_pidfd.return_value = False
         watcher = policy.get_child_watcher()
         self.assertIsInstance(watcher, asyncio.ThreadedChildWatcher)
 
         self.assertIs(policy._watcher, watcher)
 
         self.assertIs(watcher, policy.get_child_watcher())
 
+        policy = self.create_policy()
+        self.assertIsNone(policy._watcher)
+        unix_events.can_use_pidfd = mock.Mock()
+        unix_events.can_use_pidfd.return_value = True
+        watcher = policy.get_child_watcher()
+        self.assertIsInstance(watcher, asyncio.PidfdChildWatcher)
+
+        self.assertIs(policy._watcher, watcher)
+
+        self.assertIs(watcher, policy.get_child_watcher())
+
     def test_get_child_watcher_after_set(self):
         policy = self.create_policy()
         watcher = asyncio.FastChildWatcher()