Skip to content

bpo-36607: Eliminate RuntimeError raised by asyncio.all_tasks() #13971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 11, 2019
Merged

bpo-36607: Eliminate RuntimeError raised by asyncio.all_tasks() #13971

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e7f28d9
Eliminate RuntimeError raised by asyncio.all_tasks()
asvetlov Jun 11, 2019
eb3d2a3
Address comments
asvetlov Jun 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions Lib/asyncio/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,19 @@ def all_tasks(loop=None):
"""Return a set of all tasks for the loop."""
if loop is None:
loop = events.get_running_loop()
# NB: set(_all_tasks) is required to protect
# NB: list(_all_tasks) is required to protect
# from https://bugs.python.org/issue34970 bug
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write a proper comment here without referencing an issue on bpo? I mean a url is fine, but it requires the reader to navigate to it in order to understand what's going on.

return {t for t in list(_all_tasks)
# NB: Have to repeat on RuntimeError, other thread
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no point in prefixing all comments with "NB"; it's just a regular comment. I'd suggest to write something along the following (as it took me sometime to understand why this change is necessary at all):

"Looping over a WeakSet (_all_tasks) isn't safe as it can be updated from another thread while we do so. Therefore we cast it to list prior to filtering. The list cast itself requires iteration, so we repeat it several times ignoring RuntimeErrors (which are not very likely to occur). See issues 34970 and 36607 for details."

# can modify _all_tasks during list(_all_tasks) call
# https://bugs.python.org/issue36607
while True:
try:
tasks = list(_all_tasks)
except RuntimeError:
pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like unchecked loops like that. There must be a retry count or something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The number of iterations is the question.
What is good enough to prevent a collision? 10? 100? 1000?
I think the number depends on count of threads and a number of tasks inside every processed loop.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say 1000 should be enough. It's high enough to give it enough attempts and it's low enough to abort early if something weird is going on.

else:
break
return {t for t in tasks
if futures._get_loop(t) is loop and not t.done()}


Expand All @@ -54,9 +64,19 @@ def _all_tasks_compat(loop=None):
# method.
if loop is None:
loop = events.get_event_loop()
# NB: set(_all_tasks) is required to protect
# NB: list(_all_tasks) is required to protect
# from https://bugs.python.org/issue34970 bug
return {t for t in list(_all_tasks) if futures._get_loop(t) is loop}
# NB: Have to repeat on RuntimeError, other thread
# can modify _all_tasks during list(_all_tasks) call
# https://bugs.python.org/issue36607
while True:
try:
tasks = list(_all_tasks)
except RuntimeError:
pass
else:
break
return {t for t in tasks if futures._get_loop(t) is loop}


def _set_task_name(task, name):
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2019-06-11-13-52-04.bpo-36607.5_mJkQ.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Eliminate :exc:`RuntimeError` raised by :func:`asyncio.all_tasks()` if
internal tasks weak set is changed by another thread during iteration.