Runners framework

[ben-denham]

• Set up a framework for RunnerBackend classes that can implement different strategies for executing tasks (laying the groundwork for distributed task execution with Ray)
• Implement spawn and fork backends to cover existing process-based execution paradigms.
• The process-based backends no longer rely on concurrent.futures, which made use of threading that required all subprocesses to be forked up front before any task execution. By no longer relying concurrent.futures, the fork backend can use shared/forked memory to share the results of earlier tasks with later tasks. Also, the fork backend now implicitly uses shared memory to share task context with subprocesses.

• Fix TaskDiedError when keyboard-interrupting a fork runner.
• Introduce explicit runner_backend='serial' instead of implicitly switching to serial processing when max_workers=1 #31

[ben-denham]

Notes from development:

• Mac and Windows don't support fork()
• fork() will no longer be the default on Linux from Python 3.14 (multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' python/cpython#84559), but we should still be able to continue using it.
• It is unclear whether sub-interpreter Python API support from https://peps.python.org/pep-0734/ (Python 3.13?) will be useful, as it looks like direct memory sharing may still be limited.
  • Some discussions suggest there could be options added for memory sharing: https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855
• forkserver doesn't look like a feasible solution, as getting objects into the spawned parent process that is forked from can only be achieved by specifying modules to load with set_forkserver_preload().

Plan:

• Mac and Windows will use spawn, where memory must be copied from parent to child and from child to parent.
• Linux to default to fork, where memory can be shared from parent to child, but must still be copied back from child to parent.
  • We will still make this the default even after Python 3.14, given our use case merits it.
  • CUDA doesn't support fork, so spawn or Ray would need to be used: https://pytorch.org/docs/stable/notes/multiprocessing.html
• Re-consider potential benefits of subinterpreters when they become available.