Use Py_SETREF() macro to fix race conditions (potential race conditions) in C extensions

[vstinner]

A lot of C code in Python uses this pattern:

Py_XDECREF(var);
var = value;

If var was holding the last strong reference to a Python object, the object finalizer is called which can call arbitrary Python code which can access the var variable which is now a dangling pointer, and so Python does just crash in this case.

The correct pattern is:

old_var = var;
var = value;
Py_XDECREF(old_var);

And Py_SETREF() can be used to make this code shorter and easier to read and maintain:

Py_XSETREF(var, value);

While the pattern is unsafe, maybe it's not possible to crash Python. But I prefer to replace this pattern with Py_SETREF() or Py_XSETREF() so I don't have to think about: oh, is it possible to crash Python?

Again, Py_SETREF() also makes the code shorter, easier to read and maintain.

While we are here, I also propose to replace:

Py_XDECREF(var);
var = NULL;

with:

Py_CLEAR(var);

which is shorter than:

Py_XSETREF(var, NULL);

---

See also issue #99300 "Replace Py_INCREF()/Py_XINCREF() usage with Py_NewRef()/Py_XNewRef()" and issue #99481 "Add Py_HOLD_REF() macro to hold a strong reference to a Python object while executing code".

Linked PRs

• gh-99537: Use Py_SETREF() and Py_CLEAR() functions in C code #99538
• gh-99537: Use Py_SETREF() function in long C code #99655
• gh-99537: Use Py_SETREF() function in C code #99656
• gh-99537: Use Py_SETREF() function in C code #99657
• gh-99537: Use Py_CLEAR() function in C code #99686
• gh-99537: Use Py_SETREF(var, NULL) in C code #99687

[vstinner]

I wrote an implementation of this idea: PR #99538.

[vstinner]

See also #98724

[vstinner]

Ok, I converted all patterns to Py_SETREF(), Py_XSETREF() or Py_CLEAR(). I close the issue. Thanks for reviews!

Quick comparison with git grep -E 'Py_X?SETREF'|wc -l:

• Python 3.11: 294 lines
• Python 3.12: 458 lines (1.6x more)