Deadlock at process shutdown #54918
Labels
confirmed-bug
Issues with confirmed bugs.
help wanted
Issues that need assistance from volunteers or PRs that need help to proceed.
v8 engine
Issues and PRs related to the V8 dependency.
Comments
|
As written in #52550 (comment), applying the patch from #47452 seems to fix the issue, but it is not a solution since that patch brings other issues. |
|
This can been seen in various CI runs, assuming confirmed-bug |
|
The output of |
|
@aduh95 see #54918 (comment). It seems the same patch. |
|
No it's not, it's based on it but it's meant to fix the test failures. |
|
The issue persists 😞 Also, |
avivkeller
added
the
help wanted
|
Hey, I've added the
help wanted
|
|
From my testing, the following snippet hangs occassionally, which might be related to this?: Line 640 in 2545b9e
Some task isn't completing (but that was probably already known), do we know what task, or how to find out? Update 1 It seems a task is getting stuck during
Update 4 It's not a catch-22, because the hanging process appears to be hanging before the process destruction would even occur. I suspect that if you Indeed, our hanging worker thread appears to be: Thread 8 (Thread 0x7fb7c74006c0 (LWP 1322244) "node"):
#0 0x00007fb7cd49e22e in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a4a5eefdc8) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55a4a5eefdc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007fb7cd49e2ab in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55a4a5eefdc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fb7cd4a0990 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a4a5eefd78, cond=0x55a4a5eefda0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x55a4a5eefda0, mutex=0x55a4a5eefd78) at ./nptl/pthread_cond_wait.c:618
#5 0x000055a46a13ad8c in void heap::base::Stack::SetMarkerForBackgroundThreadAndCallbackImpl<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*)::{lambda()#1}>(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*)::{lambda()#1})::{lambda()#1}>(heap::base::Stack*, void*, void const*) ()
#6 0x000055a46acb27e3 in PushAllRegistersAndIterateStack ()
#7 0x000055a46a13b2df in v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*) ()
#8 0x000055a46a1ad972 in v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) ()
#9 0x000055a46a1ae0a8 in v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) ()
#10 0x000055a46a1e4891 in v8::internal::LocalFactory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) ()
#11 0x000055a46a17a4fe in v8::internal::FactoryBase<v8::internal::LocalFactory>::NewProtectedFixedArray(int) ()
#12 0x000055a46a364a6d in v8::internal::DeoptimizationData::New(v8::internal::LocalIsolate*, int) ()
#13 0x000055a46a7b8d3f in v8::internal::maglev::MaglevCodeGenerator::GenerateDeoptimizationData(v8::internal::LocalIsolate*) ()
#14 0x000055a46a7b991b in v8::internal::maglev::MaglevCodeGenerator::BuildCodeObject(v8::internal::LocalIsolate*) [clone .part.0] ()
#15 0x000055a46a7d9948 in v8::internal::maglev::MaglevCodeGenerator::Assemble() ()
#16 0x000055a46a82f269 in v8::internal::maglev::MaglevCompiler::Compile(v8::internal::LocalIsolate*, v8::internal::maglev::MaglevCompilationInfo*) ()
#17 0x000055a46a830b49 in v8::internal::maglev::MaglevCompilationJob::ExecuteJobImpl(v8::internal::RuntimeCallStats*, v8::internal::LocalIsolate*) ()
#18 0x000055a469ffb6bb in v8::internal::OptimizedCompilationJob::ExecuteJob(v8::internal::RuntimeCallStats*, v8::internal::LocalIsolate*) ()
#19 0x000055a46a831047 in v8::internal::maglev::MaglevConcurrentDispatcher::JobTask::Run(v8::JobDelegate*) ()
#20 0x000055a46b21b45f in v8::platform::DefaultJobWorker::Run() ()
#21 0x000055a469cb125f in node::(anonymous namespace)::PlatformWorkerThread(void*) ()
#22 0x00007fb7cd4a1732 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#23 0x00007fb7cd51c2b8 in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78Update 5 The hanging tasks is pushed via the following bt: So it looks like this is a V8 hanging-task issue? I'll reach out to them and see if they know anything. (https://issues.chromium.org/issues/374285493) |
dubzzz
added a commit
to dubzzz/fast-check
that referenced
this issue
Oct 29, 2024
Retries are needed for now against Node 23. There is an open ticket on V8 side (see https://issues.chromium.org/issues/374285493) and one on Node side (see nodejs/node#54918) regarding hanging issues of Node 23.
|
From the discussion in https://issues.chromium.org/issues/374285493 (closed as Won't Fix - Intended Behavior) the issue is not in V8 but in Node.js. |
This was referenced Nov 27, 2024
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is the same as other tests that time out. Refs: nodejs#54918
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is likely the same as other tests that time out. Refs: nodejs#54918
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is likely the same as other tests that time out. Refs: nodejs#54918 Refs: nodejs#54844
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is the same as other tests that time out. Refs: nodejs#54918 Refs: nodejs#54802
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is likely the same as other tests that time out. Refs: nodejs#54918 Refs: nodejs#54802
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
The issue is likely the same as other tests that time out. Refs: nodejs#54918 Refs: nodejs#54844 Refs: nodejs#54802
lpinca
added a commit
to lpinca/node
that referenced
this issue
Nov 28, 2024
Move `test/sequential/test-worker-arraybuffer-zerofill.js` back to `test/parallel/test-worker-arraybuffer-zerofill.js` and remove the flaky designation. The original issue is likely the same as other tests that time out. Refs: nodejs#54918 Refs: nodejs#54839 Refs: nodejs#54802
nodejs-github-bot
pushed a commit
that referenced
this issue
May 4, 2025
According to the documentation, the v8 tasks should be executed based on priority. Previously we always execute the tasks in FIFO order, this changes the NodePlatform implementation to execute the higher priority tasks first. The tasks used to schedule timers for the delayed tasks are run in FIFO order since priority is irrelavent for the timer scheduling part while the tasks unwrapped by the timer callbacks are still ordered by priority. PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
nodejs-github-bot
pushed a commit
that referenced
this issue
May 4, 2025
we should not be blocking on the worker tasks on the main thread in one go. Doing so leads to two problems: 1. If any of the worker tasks post another foreground task and wait for it to complete, and that foreground task is posted right after we flush the foreground task queue and before the foreground thread goes into sleep, we'll never be able to wake up to execute that foreground task and in turn the worker task will never complete, and we have a deadlock. 2. Worker tasks can be posted from any thread, not necessarily associated with the current isolate, and we can be blocking on a worker task that is associated with a completely unrelated isolate in the event loop. This is suboptimal. However, not blocking on the worker tasks at all can lead to loss of some critical user-blocking worker tasks e.g. wasm async compilation tasks, which should block the main thread until they are completed, as the documentation suggets. As a compromise, we currently only block on user-blocking tasks to reduce the chance of deadlocks while making sure that criticl user-blocking tasks are not lost. PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
|
I changed #58047 to only ref this issue instead of closing it, because I think that PR only eliminates a subset of the deadlocks, but not all - it may still be possible for a user blocking worker task to post a foreground task right before the foreground thread goes into sleep, just that most user blocking worker tasks tend not do that (but it is still possible e.g. with the use of the high priority concurrent compilation flags, or a major GC triggered by the heap reaching a hard limit) or that they are less likely to hit that unfortunate timing. So I left a big |
RafaelGSS
pushed a commit
that referenced
this issue
May 5, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
RafaelGSS
pushed a commit
that referenced
this issue
May 5, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
RafaelGSS
pushed a commit
that referenced
this issue
May 5, 2025
According to the documentation, the v8 tasks should be executed based on priority. Previously we always execute the tasks in FIFO order, this changes the NodePlatform implementation to execute the higher priority tasks first. The tasks used to schedule timers for the delayed tasks are run in FIFO order since priority is irrelavent for the timer scheduling part while the tasks unwrapped by the timer callbacks are still ordered by priority. PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
RafaelGSS
pushed a commit
that referenced
this issue
May 5, 2025
we should not be blocking on the worker tasks on the main thread in one go. Doing so leads to two problems: 1. If any of the worker tasks post another foreground task and wait for it to complete, and that foreground task is posted right after we flush the foreground task queue and before the foreground thread goes into sleep, we'll never be able to wake up to execute that foreground task and in turn the worker task will never complete, and we have a deadlock. 2. Worker tasks can be posted from any thread, not necessarily associated with the current isolate, and we can be blocking on a worker task that is associated with a completely unrelated isolate in the event loop. This is suboptimal. However, not blocking on the worker tasks at all can lead to loss of some critical user-blocking worker tasks e.g. wasm async compilation tasks, which should block the main thread until they are completed, as the documentation suggets. As a compromise, we currently only block on user-blocking tasks to reduce the chance of deadlocks while making sure that criticl user-blocking tasks are not lost. PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 6, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 6, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 6, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 6, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
RafaelGSS
pushed a commit
that referenced
this issue
May 14, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
RafaelGSS
pushed a commit
that referenced
this issue
May 14, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 16, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 16, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 17, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 18, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 18, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 19, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
aduh95
pushed a commit
that referenced
this issue
May 19, 2025
PR-URL: #58047 Refs: #47452 Refs: #54918 Reviewed-By: Stephen Belanger <[email protected]>
mhdawson
removed
the
tsc-agenda
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Version
v23.0.0-pre
Platform
Subsystem
No response
What steps will reproduce the bug?
There is a deadlock that prevents the Node.js process from exiting. The issue is causing a lot (all?) of timeout failures in our CI. It can be reproduced by running a test in parallel with our
test.pytool, for exampleSee also
test-stream-readable-unpipe-resume#54133 (comment)test-net-write-fully-async-bufferas flaky #52959 (comment)How often does it reproduce? Is there a required condition?
Rarely, but often enough to be a serious issue for CI.
What is the expected behavior? Why is that the expected behavior?
The process exits.
What do you see instead?
The process does not exit.
Additional information
Attaching
gdbto two of the hanging processes obtained from the command above, produces the following outputs:The text was updated successfully, but these errors were encountered: