fix: support Python 3.14 #5646

henryiii · 2025-05-08T18:47:47Z

Description

Let's try beta 1!

Suggested changelog entry:

* Support Python 3.14 (beta 1)

.github/workflows/ci.yml

henryiii · 2025-05-08T19:36:21Z

FAILED test_methods_and_attributes.py::test_dynamic_attributes - AssertionError: assert not True
 +  where True = hasattr(<pybind11_tests.methods_and_attributes.DynamicClass object at 0x7f3c250dd750>, 'foo')
FAILED test_operator_overloading.py::test_return_set_of_unhashable - assert False
 +  where False = <built-in method startswith of str object at 0x7f3c252b3030>('unhashable type:')
 +    where <built-in method startswith of str object at 0x7f3c252b3030> = "cannot use 'pybind11_tests.operators.HashMe' as a set element (unhashable type: 'pybind11_tests.operators.HashMe')".startswith
 +      where "cannot use 'pybind11_tests.operators.HashMe' as a set element (unhashable type: 'pybind11_tests.operators.HashMe')" = str(TypeError("cannot use 'pybind11_tests.operators.HashMe' as a set element (unhashable type: 'pybind11_tests.operators.HashMe')"))
 +        where TypeError("cannot use 'pybind11_tests.operators.HashMe' as a set element (unhashable type: 'pybind11_tests.operators.HashMe')") = TypeError('Unable to convert function return value to a Python type! The signature was\n\t() -> set[pybind11_tests.operators.HashMe]').__cause__
 +          where TypeError('Unable to convert function return value to a Python type! The signature was\n\t() -> set[pybind11_tests.operators.HashMe]') = <ExceptionInfo TypeError('Unable to convert function return value to a Python type! The signature was\n\t() -> set[pybind11_tests.operators.HashMe]') tblen=1>.value
FAILED test_pickling.py::test_roundtrip_with_dict[PickleableWithDict] - AttributeError: 'pybind11_tests.pickling.PickleableWithDict' object has no attribute 'dynamic'
FAILED test_pickling.py::test_roundtrip_with_dict[PickleableWithDictNew] - AttributeError: 'pybind11_tests.pickling.PickleableWithDictNew' object has no attribute 'dynamic'

rwgk · 2025-05-08T21:03:55Z

The tests/test_operator_overloading.py failure is a trivial fix, but I'm not sure about the other two. They could easily steal a couple hours each to take care of...

henryiii · 2025-05-09T06:04:50Z

Looks like two errors: py::hash isn’t working, and dynamic attributes are broken. @vstinner I don’t see anything in https://docs.python.org/3.14/whatsnew/3.14.html#id11 that would seem to indicate changes here.

I can try bisecting later.

vstinner · 2025-05-09T06:12:14Z

where <built-in method startswith of str object at 0x7f3c252b3030> = "cannot use 'pybind11_tests.operators.HashMe' as a set element (unhashable type: 'pybind11_tests.operators.HashMe')".startswith

Yeah, I changed the error message in: python/cpython#132825

You may be able to fix your test by replacing "startswith" with "contains" ("in").

henryiii · 2025-05-09T16:04:10Z

Wow, that's a huge improvement:

Love the syntax highlighting in the REPL now, too. :)

I thought there was another issue with py::hash, but I see it's just that, so it's only the dynamic change now. I still don't see anything in the upgrade guide. But one issue should be easy to bisect.

vstinner · 2025-05-09T16:20:36Z

I still don't see anything in the upgrade guide

The change was documented 1h ago: python/cpython@de28651.

henryiii · 2025-05-09T16:26:27Z

BTW, we are also running into https://gitlab.kitware.com/cmake/cmake/-/issues/26926 on Windows, CMake is confused and trying to use a free-threaded lib. I've seen that with scikit-build-core, too, in scikit-build/scikit-build-core#1074.

henryiii · 2025-05-09T21:22:13Z

Bisecting this shows it was broken by python/cpython#123192. I've commented on python/cpython#115776.

Edit: opened python/cpython#133912

Signed-off-by: Henry Schreiner <[email protected]>

b-pass · 2025-05-15T21:17:31Z

tests/test_multiple_interpreters.py

+    if sys.version_info >= (3, 15):
        import interpreters
-    elif sys.version_info >= (3, 13):
+    elif sys.version_info >= (3, 14):
        import _interpreters as interpreters


Looks like they didn't do this rename in 3.14, so it should be _interpreters for 3.13 and 3.14. Maybe just get rid of the 3.15 and leave it as :

if sys.version_info >= (3, 13): import _interpreters as interpreters

Or future proof it as:

if sys.version_info >= (3, 13): try: import interpreters except ImportError: import _interpreters as interpreters

I think there will be a pypi package for 3.14 eventually (though I think it was supposed to come to 3.13 originally, so who knows...), but let's just leave it as <3.15 for now.

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-05-16T06:09:16Z

Is sys._is_immortal not working on Windows? It seems to be returning False. Also, I see a warning about the GIL being engaged due to a GIL extension we should investigate.

henryiii · 2025-05-16T06:23:48Z

It's not Windows specific, all of them report:

sys._is_immortal(cls)=False
sys.getrefcount(cls)=1152921504606846979

And exo_planet_c_api is re-engaging the GIL.

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-05-16T14:26:47Z

Can be reproduced in pure Python on 3.14t:

>>> import sys
>>> class Thing:
...     pass
>>> sys._is_immortal(Thing)
False
>>> sys.getrefcount(Thing)
1152921504606846980

Let's just ignore large (30 bits+) values.

colesbury · 2025-05-16T14:39:20Z

In the 3.14 free threaded build, types, module-level functions, and some other objects use deferred reference counting. They have a large value (~~2^62 or something~~ 2^60) added to their reference count so that it doesn't hit zero. These objects are not immortal; they can be collected during GC.

colesbury · 2025-05-16T14:50:57Z

There are some more details here: https://peps.python.org/pep-0703/#deferred-reference-counting

There's a few things going on here:

Heap type objects aren't immortal in 3.14t, but they were immortal as a temporary measure in 3.13t
Heap type objects use a combination of deferred reference counting and per-thread reference counting. PEP 703 has a description of both. They have a very large reference count, but unlike immortal objects, the reference count isn't fixed and the objects can eventually be collected.
Instances don't directly incref their types in 3.14t ¹ in order to avoid scaling bottlenecks due to reference count contention. They instead increment a counter in a per-thread array corresponding to the type. These "per-thread" reference counts aren't included in Py_REFCNT or sys.getrefcount() because safely doing so would require a stop-the-world pause. The GC takes them into consideration when deciding whether it can free a heap type object.

This is specifically aboutPyObject_Init(). If you manually call Py_INCREF(type_obj) on a heap type object, it will directly modify the type's reference count. ↩

henryiii · 2025-05-16T15:48:07Z

The test is just making sure pybind11 classes refcount correctly. I think it's fine for it to be checked to be either the correct value or a large value. Unless there's a way to "stop the world" and get the total recount?

henryiii · 2025-05-16T15:50:39Z

Subinterpreter issue on Windows 3.14t (CC @b-pass):

  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'widget_module', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
  
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  test_embed.exe is a Catch v2.13.10 host application.
  Run with -? for options
  
  -------------------------------------------------------------------------------
  Subinterpreter
  -------------------------------------------------------------------------------
  D:\a\pybind11\pybind11\tests\test_embed\test_interpreter.cpp(339)
  ...............................................................................
  
  D:\a\pybind11\pybind11\tests\test_embed\test_interpreter.cpp(339): FAILED:
    {Unknown expression after the reported line}
  due to unexpected exception with message:
    ImportError: module widget_module does not support loading in subinterpreters
  
  Assertion failed: t_int == main_int, file D:\a\pybind11\pybind11\tests\test_embed\test_interpreter.cpp, line 498

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-05-16T16:47:38Z

Our single free-threaded test before didn't have support for embedded tests, so that's why this slipped in, I expect. I'm adding all the free-threaded Pythons to the CI.

henryiii · 2025-05-16T19:51:35Z

In the free threading build for 3.13t and 3.14t, this isn't working:

        // widget_module did not provide the mod_per_interpreter_gil tag, so it cannot be imported
        bool caught = false;
        try {
            py::module_::import("widget_module");
        } catch (pybind11::error_already_set &pe) {
            T_REQUIRE(pe.matches(PyExc_ImportError));
            std::string msg(pe.what());
            T_REQUIRE(msg.find("does not support loading in subinterpreters")
                      != std::string::npos);
            caught = true;
        }
        T_REQUIRE(caught);

Though it's clearly printing out:

Subinterpreter
-------------------------------------------------------------------------------
/home/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:339
...............................................................................

/home/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:339: FAILED:
  {Unknown expression after the reported line}
due to unexpected exception with message:
  ImportError: module widget_module does not support loading in subinterpreters

-------------------------------------------------------------------------------
Per-Subinterpreter GIL
-------------------------------------------------------------------------------
/home/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:470
...............................................................................

Is something different about catching exceptions in the free-threading build?

macOS (both) and ubuntu 3.13t fail. ubuntu 3.14t and Windows (both) hang.

b-pass · 2025-05-16T20:17:17Z

In the free threading build for 3.13t and 3.14t, this isn't working:

[...]

Is something different about catching exceptions in the free-threading build?

macOS (both) and ubuntu 3.13t fail. ubuntu 3.14t and Windows (both) hang.

I think the issue is concurrent calls to the module init for the embedded modules. I think the below diff would fix it. I can make a separate PR for this if you want. I am working on another PR that includes for switching PYBIND11_EMBEDDED_MODULE to multiphase init (so that it can support py::mod_gil_not_used() and py::multiple_interpreters), that might be ready to start review soon....

I don't have any easy way to test this in 3.14t locally, sorry

diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h
index a456e80a..b7bd3bec 100644
--- a/include/pybind11/embed.h
+++ b/include/pybind11/embed.h
@@ -41,15 +41,18 @@
 #define PYBIND11_EMBEDDED_MODULE(name, variable)                                                  \
     static ::pybind11::module_::module_def PYBIND11_CONCAT(pybind11_module_def_, name);           \
     static void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &);                     \
-    static PyObject PYBIND11_CONCAT(*pybind11_init_wrapper_, name)() {                            \
-        auto m = ::pybind11::module_::create_extension_module(                                    \
-            PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name));      \
-        try {                                                                                     \
-            PYBIND11_CONCAT(pybind11_init_, name)(m);                                             \
-            return m.ptr();                                                                       \
-        }                                                                                         \
-        PYBIND11_CATCH_INIT_EXCEPTIONS                                                            \
-        return nullptr;                                                                           \
+    static PyObject *PYBIND11_CONCAT(pybind11_init_wrapper_, name)() {                            \
+        static auto result = []() -> PyObject * {                                                 \
+            auto m = ::pybind11::module_::create_extension_module(                                \
+                PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name));  \
+            try {                                                                                 \
+                PYBIND11_CONCAT(pybind11_init_, name)(m);                                         \
+                return m.ptr();                                                                   \
+            }                                                                                     \
+            PYBIND11_CATCH_INIT_EXCEPTIONS                                                        \
+            return nullptr;                                                                       \
+        }();                                                                                      \
+        return result;                                                                            \
     }                                                                                             \
     PYBIND11_EMBEDDED_MODULE_IMPL(name)                                                           \
     ::pybind11::detail::embedded_module PYBIND11_CONCAT(pybind11_module_, name)(                  \

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-05-16T22:23:00Z

I'm low on battery, so I'll drop the free threaded ci and that can be a separate PR. I can try things locally normally but that will finish my battery. ;) I tried your patch but it looks like it's failing.

Sounds good.

This reverts commit c4226a0.

Signed-off-by: Henry Schreiner <[email protected]>

include/pybind11/pytypes.h

rwgk · 2025-05-17T00:00:49Z

tests/test_multiple_interpreters.py

@@ -86,7 +86,7 @@ def test_dependent_subinterpreters():

    sys.path.append(".")

-    if sys.version_info >= (3, 14):
+    if sys.version_info >= (3, 15):


Is this final for 3.14, or could this still change before the 3.14.0 release?

If the latter: maybe add a comment?

It will continue to work. I think intepreters is supposed to be released on PyPI during the 3.14 lifecycle.

Signed-off-by: Henry Schreiner <[email protected]>

include/pybind11/pytypes.h

rwgk reviewed May 8, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

henryiii force-pushed the henryiii/chore/py314 branch from 1a879b5 to 63877be Compare May 11, 2025 04:40

This was referenced May 12, 2025

Regression in 3.14: setting __dict__ on custom type broken (mypyc, pybind11) python/cpython#133912

Closed

ci: bump default versions #5657

Merged

fix: use original dict #5658

Merged

henryiii force-pushed the henryiii/chore/py314 branch 3 times, most recently from 4ff28e2 to 574a024 Compare May 13, 2025 18:49

henryiii mentioned this pull request May 15, 2025

feat: support for sub-interpreters #5564

Merged

henryiii added 6 commits May 15, 2025 14:42

ci: support Python 3.14

77f03f4

Signed-off-by: Henry Schreiner <[email protected]>

fix: Python 3.14 name change

3bd75bd

Signed-off-by: Henry Schreiner <[email protected]>

tests: fix expected output to handle Python 3.14

236b32f

Signed-off-by: Henry Schreiner <[email protected]>

fix: tighten CLI and add color on 3.14+

ce693fd

Signed-off-by: Henry Schreiner <[email protected]>

tests: ignore failure on 3.14.0b1

cd5125c

Signed-off-by: Henry Schreiner <[email protected]>

fix: support Python 3.14.0b1 with interperters

2cd17ba

Signed-off-by: Henry Schreiner <[email protected]>

henryiii force-pushed the henryiii/chore/py314 branch from 0e89da3 to 2cd17ba Compare May 15, 2025 18:43

b-pass reviewed May 15, 2025

View reviewed changes

henryiii and others added 5 commits May 15, 2025 17:23

Update test_multiple_interpreters.py

8e4575a

Update test_multiple_interpreters.py

5e0324c

fix: new breakage for 3.14 fixed

76408fd

Signed-off-by: Henry Schreiner <[email protected]>

fix: handle empty annotations 3.14

c5699a8

Signed-off-by: Henry Schreiner <[email protected]>

fix: Python 3.14 may not create the annotations dict

8031427

Signed-off-by: Henry Schreiner <[email protected]>

henryiii marked this pull request as draft May 16, 2025 06:18

tests: ignore large values for refcount too

1f2e5d6

Signed-off-by: Henry Schreiner <[email protected]>

henryiii force-pushed the henryiii/chore/py314 branch from 7d3aefb to 1f2e5d6 Compare May 16, 2025 14:24

style: pre-commit fixes

6caea71

ci: enable all free-threaded builds

6e9c304

Signed-off-by: Henry Schreiner <[email protected]>

fix: patch for embed

c4226a0

Signed-off-by: Henry Schreiner <[email protected]>

henryiii added 2 commits May 16, 2025 18:23

Revert "fix: patch for embed"

151f225

This reverts commit c4226a0.

ci: drop new 3.xt additions

7bca953

Signed-off-by: Henry Schreiner <[email protected]>

henryiii marked this pull request as ready for review May 16, 2025 23:41

rwgk approved these changes May 17, 2025

View reviewed changes

b-pass mentioned this pull request May 17, 2025

feat: change PYBIND11_EMBEDDED_MODULE to multiphase init #5665

Merged

fix: logic issue, also add some comments

87e7a50

Signed-off-by: Henry Schreiner <[email protected]>

henryiii commented May 17, 2025

View reviewed changes

include/pybind11/pytypes.h Outdated Show resolved Hide resolved

Update include/pybind11/pytypes.h

7a4e2db

henryiii mentioned this pull request May 17, 2025

feat: Embeded sub-interpreters #5666

Merged

henryiii merged commit 094343c into pybind:master May 17, 2025
66 of 67 checks passed

henryiii deleted the henryiii/chore/py314 branch May 17, 2025 01:58

github-actions bot added the needs changelog Possibly needs a changelog entry label May 17, 2025

henryiii removed the needs changelog Possibly needs a changelog entry label May 17, 2025

fix: support Python 3.14 #5646

fix: support Python 3.14 #5646

Uh oh!

Conversation

henryiii commented May 8, 2025

Description

Suggested changelog entry:

Uh oh!

Uh oh!

henryiii commented May 8, 2025

Uh oh!

rwgk commented May 8, 2025

Uh oh!

henryiii commented May 9, 2025

Uh oh!

vstinner commented May 9, 2025

Uh oh!

henryiii commented May 9, 2025

Uh oh!

vstinner commented May 9, 2025

Uh oh!

henryiii commented May 9, 2025

Uh oh!

henryiii commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-pass May 15, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii May 16, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii commented May 16, 2025

Uh oh!

henryiii commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented May 16, 2025

Uh oh!

colesbury commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colesbury commented May 16, 2025

Footnotes

Uh oh!

henryiii commented May 16, 2025

Uh oh!

henryiii commented May 16, 2025

Uh oh!

henryiii commented May 16, 2025

Uh oh!

henryiii commented May 16, 2025

Uh oh!

b-pass commented May 16, 2025

Uh oh!

henryiii commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rwgk May 17, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii May 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henryiii commented May 9, 2025 •

edited

Loading

henryiii commented May 16, 2025 •

edited

Loading

colesbury commented May 16, 2025 •

edited

Loading

henryiii commented May 16, 2025 •

edited

Loading