Issues with Multiprocess and AsyncRunner in adaptive for Phase Diagram Illustration

[hz-xiaxz]

Hello! I'm currently using the adaptive library to depict a phase diagram. I'm attempting to leverage multiple cores on my server by employing multiprocessing. However, I've encountered issues when using the AsyncRunner.

Below is the code snippet that I'm running in a single Python script:

import adaptive
from concurrent.futures import ProcessPoolExecutor
from functools import partial

def sample(xy, data):
    # Implementation of time-demanding function
    # ...

learner = adaptive.Learner2D(partial(sample, data=data), bounds=((0, 7), (7, 13)))

with ProcessPoolExecutor(max_workers=96) as executor:
    runner = adaptive.Runner(learner, executor=executor, npoints_goal=10)
    runner.task.print_stack()

It's important to note that the sample function is computationally intensive, taking hours to complete. Each function instance writes to its own identical files, preventing conflicts.

However, executing the code above results in an incomplete run, leaving the stack trace below:

Stack for <Task pending name='Task-1' coro=<AsyncRunner._run() running at /xxx/envs/FFS/lib/python3.10/site-packages/adaptive/runner.py:771>> (most recent call last):
  File "/xxx/envs/FFS/lib/python3.10/site-packages/adaptive/runner.py", line 771, in _run
    async def _run(self) -> None:

While using the BlockingRunner class works fine, I suspect there might be an efficiency bottleneck in the multiprocessing process.

Could you please help me identify the problem in my script? Additionally, do you think it would be more efficient to use the AsyncRunner in my case?

Thanks a lot for your assistance!

[basnijholt]

The Runner will manage and close the executor for you, so you shouldn't use the context manager (with).

Try:

import adaptive
from concurrent.futures import ProcessPoolExecutor
from functools import partial

def sample(xy, data):
    # Implementation of time-demanding function
    # ...

learner = adaptive.Learner2D(partial(sample, data=data), bounds=((0, 7), (7, 13)))

executor = ProcessPoolExecutor(max_workers=96)
runner = adaptive.Runner(learner, executor=executor, npoints_goal=10)
runner.task.print_stack()

The runner is non blocking, so the status message you are seeing is expected and not an actual error message. It shows where in the code the runner currently is, and immediately afterwards it will close the executor (when reaching the end of your with: block).

The AsyncRunner is useful whenever you are in an interactive development environment such as a notebook because it don't block the notebook kernel. If you are using a script you can still use runner.ioloop.run_until_complete(runner.task) to block until it is done (and soon use #444).

[hz-xiaxz]

That solves my problem, thanks a lot!