Not able to start an instantiated Thread in a child process through multiprocessing.Process in python 3.13

[anadrianmanrique]

Bug report

Bug description:

I'm experiencing an issue specifically in python 3.13 regarding Threads in the context of multiprocessing.
The following code is working in python < 3.13

from multiprocessing import Process
from threading import Thread
import time
class MyThread(Thread):
    def __init__(self):
        Thread.__init__(self)
        self.__data = ''

    def run(self):
        print("Hi from thread")
        print(self.__data)


class Aclass():
    def __init__(self):
        self._t = MyThread()
        self._t.daemon = True

    def start(self):
        self._t.start()
        print("thread started")

if __name__ == '__main__':
    t = Aclass()
    p = Process(target = t.start )
    p.start()
    time.sleep(2)

After executing the above in python 3.13 I get this output:

 File "/usr/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/impacket/master/impacket/tests/SMB_RPC/test_bug_process_thread.py", line 18, in start
    self._t.start()
    ~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/threading.py", line 973, in start
    _start_joinable_thread(self._bootstrap, handle=self._handle,
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           daemon=self.daemon)
                           ^^^^^^^^^^^^^^^^^^^
RuntimeError: thread already started

I managed to workaround this by delaying the Thread instance initialization, moving the Thread.init() call to start() method in the derived class, so everything gets executed in the context of the child process.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-134381: Improve thread safety in _PyThread_AfterFork by preserving not-started handles #134514
• [3.14] gh-134381: Fix RuntimeError when starting not-yet started Thread after fork (gh-134514) #134596
• [3.13] gh-134381: Fix RuntimeError when starting not-yet started Thread after fork (gh-134514) #134597

[zangjiucheng]

Hi, I am a beginner contributor to the CPython project, I scan your issue and I believe this related to following function,

Modules/_threadmodule.c

void
_PyThread_AfterFork(struct _pythread_runtime_state *state)
{
    // gh-115035: We mark ThreadHandles as not joinable early in the child's
    // after-fork handler. We do this before calling any Python code to ensure
    // that it happens before any ThreadHandles are deallocated, such as by a
    // GC cycle.
    PyThread_ident_t current = PyThread_get_thread_ident_ex();

    struct llist_node *node;
    llist_for_each_safe(node, &state->handles) {
        ThreadHandle *handle = llist_data(node, ThreadHandle, node);
        if (handle->ident == current) {
            continue;
        }

        // Mark all threads as done. Any attempts to join or detach the
        // underlying OS thread (if any) could crash. We are the only thread;
        // it's safe to set this non-atomically.

        /* Issue Here, it will set `handle->state` to THREAD_HANDLE_DONE */
        // handle->state = THREAD_HANDLE_DONE;

        handle->once = (_PyOnceFlag){_Py_ONCE_INITIALIZED};
        handle->mutex = (PyMutex){_Py_UNLOCKED};
        _PyEvent_Notify(&handle->thread_is_exiting);
        llist_remove(node);
        remove_from_shutdown_handles(handle);
    }
}

There is a related PR to this topic: gh-114271

If I've made any mistakes, please point them out and I'll continue to explore the issue.

[ZeroIntensity]

The problem is that _PyThread_AfterFork assumes all Thread objects have started so it can remove them. This line is the problem:

cpython/Modules/_threadmodule.c

Line 302 in e6dde10

handle->state = THREAD_HANDLE_DONE;

We need to ignore threads that haven't started. That said, doing this sort of thing is incredibly weird. What's the use-case for creating a Thread in a parent process, but then starting it in a child process?

[anadrianmanrique]

In my particular scenario, I'm running tests in client/server project. Test code for the client executes server code in a Process instance, as a precondition. Some parts of the server are implemented as Thread subclasses.

[zangjiucheng]

Hi everyone, I was thinking about this today. Based on an idea from @ZeroIntensity. (It's probably a very basic implement), since this is a thread that hasn't been started yet, I can't think of any harm in keeping it. (The only trouble could be we will need another GC cycle to cleanup the dead thread)

void
_PyThread_AfterFork(struct _pythread_runtime_state *state)
{
    // gh-115035: We mark ThreadHandles as not joinable early in the child's
    // after-fork handler. We do this before calling any Python code to ensure
    // that it happens before any ThreadHandles are deallocated, such as by a
    // GC cycle.
    PyThread_ident_t current = PyThread_get_thread_ident_ex();

    struct llist_node *node;
    llist_for_each_safe(node, &state->handles) {
        ThreadHandle *handle = llist_data(node, ThreadHandle, node);
        if (handle->ident == current) {
            continue;
        }

+      // keep not_start handles – they are safe to start in the child
+      if (handle->state == THREAD_HANDLE_NOT_STARTED){
+          continue;
+      }

        // Mark all threads as done. Any attempts to join or detach the
        // underlying OS thread (if any) could crash. We are the only thread;
        // it's safe to set this non-atomically.
        handle->state = THREAD_HANDLE_DONE;
        handle->once = (_PyOnceFlag){_Py_ONCE_INITIALIZED};
        handle->mutex = (PyMutex){_Py_UNLOCKED};
        _PyEvent_Notify(&handle->thread_is_exiting);
        llist_remove(node);
        remove_from_shutdown_handles(handle);
    }
}

[ZeroIntensity]

Sounds reasonable to me.

[zangjiucheng]

Sounds reasonable to me.

Hi there, I wish to start draft a PR for this issue. Should I target to branch main or 3.13, I did more test on 3.14 / main branch. It seems another issue pop up. TypeError: cannot pickle '_thread._ThreadHandle' object.

This only fix the issue on 3.13 now, but I will also keep observe more detail about the new issue on 3.14 and main.

Thanks for your help @ZeroIntensity @anadrianmanrique

[ZeroIntensity]

Put up a PR for the main branch. If we decide to fix this for older versions, we can automatically backport the fix.

[zangjiucheng]

Done, thanks so much :)

[anadrianmanrique]

Thank you all for the quick response/fix

[colesbury]

This pattern will only work with "fork", not "spawn" because you can't pickle a Thread. In 3.13 3.14, the default start method switched from "fork" to "spawn", so this test case won't work as is. You'd need something like multiprocessing.set_start_method("fork") in the if __name__ == '__main__' in addition to the changes proposed by @zangjiucheng. It also will not work on Windows, which doesn't support "fork".

[zangjiucheng]

@colesbury Thanks a bunch! I was also wondering why this crashed. I learned something new, thanks!