Race between _PyObject_GetMethod and ensure_nonmanaged_dict

[hawkinsp]

Bug report

Bug description:

Using CPython 3.13 built at commit 2ab7e11 with TSAN enabled:

Repro:

import concurrent.futures
import functools
import threading

num_threads = 100


def closure(b, o):
  b.wait()
  o.__call__()
  o.foo = 42

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(100):
    b = threading.Barrier(num_threads)
    o = functools.partial(lambda x: x, 42)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b, o))

TSAN report:

WARNING: ThreadSanitizer: data race (pid=124621)
  Read of size 8 at 0x7f942a59fda8 by thread T86:
    #0 _PyObject_GetMethod /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1587:20 (python3.13+0x2974ea) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #1 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3744:25 (python3.13+0x3eeed3) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #2 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df4ba) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #3 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1813:12 (python3.13+0x3df4ba)
    #4 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb5ff) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #5 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x575642) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #6 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x575642)
    #7 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb273) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #8 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb273)
    #9 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb2f5) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #10 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5572) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #11 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df4ba) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #12 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1813:12 (python3.13+0x3df4ba)
    #13 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb5ff) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #14 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5cf) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #15 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5cf)
    #16 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb273) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #17 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb273)
    #18 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb2f5) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #19 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x567ca2) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #20 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c0c17) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)

  Previous atomic write of size 8 at 0x7f942a59fda8 by thread T89:
    #0 _Py_atomic_store_ptr_release /usr/local/google/home/phawkins/p/cpython/./Include/cpython/pyatomic_gcc.h:501:3 (python3.13+0x272d27) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #1 ensure_nonmanaged_dict /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:7275:9 (python3.13+0x272d27)
    #2 _PyObjectDict_SetItem /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:7311:12 (python3.13+0x272dfe) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #3 _PyObject_GenericSetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1825:19 (python3.13+0x29813e) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #4 PyObject_GenericSetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1853:12 (python3.13+0x298b17) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #5 PyObject_SetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1415:15 (python3.13+0x29543a) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:5488:27 (python3.13+0x3f4f9d) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df4ba) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1813:12 (python3.13+0x3df4ba)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb5ff) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x575642) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x575642)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb273) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb273)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb2f5) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5572) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df4ba) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1813:12 (python3.13+0x3df4ba)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb5ff) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5cf) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5cf)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb273) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb273)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb2f5) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #24 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x567ca2) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)
    #25 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c0c17) (BuildId: 7813a82c043bc03d6031bcfa367627f3fee4c283)

This is quite similar to #128100

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[ZeroIntensity]

The offending 3.13 code is here, this should be an atomic load (with acquire ordering):

cpython/Objects/object.c

Lines 1585 to 1588 in 43a2a37

	PyObject **dictptr = _PyObject_ComputedDictPointer(obj);
	if (dictptr != NULL) {
	dict = *dictptr;
	}

This was already fixed for the main branch in GH-129831, but that didn't make it to 3.13. @colesbury, do you want to backport that PR, or would you rather a fresh fix for 3.13?

[colesbury]

Thanks for tracking this down. Let's backport the PR.

[colesbury]

Backport: #130597

[colesbury]

The fix is now backported to 3.13

[ZeroIntensity]

Thanks for the fix and @hawkinsp for the heads up!