RecoverySourceHandler#runWithGenericThreadPool caused deadlock

[DaveCTurner]

We saw a benchmark of a 7.17.2 cluster get stuck with all generic threads blocked in RecoverySourceHandler#runWithGenericThreadPool (see many-shards-threaddump.txt.gz):

$ python ~/src/jstack2json/jstack2json.py many-shards-threaddump.txt | jq '.[].threads[] | select(.elasticsearch.threadpool == "generic")' -cMr | wc -l
     144
$ python ~/src/jstack2json/jstack2json.py many-shards-threaddump.txt | jq '.[].threads[] | select(.elasticsearch.threadpool == "generic") | .stack' -cMr | sed -e 's/parking to wait for  <0x[0-9a-f]*>//' | sort -u | wc -l
       1
$ python ~/src/jstack2json/jstack2json.py many-shards-threaddump.txt | jq '.[].threads[] | select(.elasticsearch.threadpool == "generic") | .stack' -cMr | head -n1 | jq '.[]' -cMr
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for  <0x0000000496000040> (a org.elasticsearch.common.util.concurrent.BaseFuture$Sync)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly([email protected]/AbstractQueuedSynchronizer.java:1047)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:243)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:45)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.runWithGenericThreadPool(RecoverySourceHandler.java:486)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$acquireSafeCommit$21(RecoverySourceHandler.java:476)
at org.elasticsearch.indices.recovery.RecoverySourceHandler$$Lambda$6704/0x0000000801a21000.run(Unknown Source)
at org.elasticsearch.index.engine.Engine$IndexCommitRef.close(Engine.java:1919)
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:74)
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:116)
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:66)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$recoverToTarget$7(RecoverySourceHandler.java:283)
at org.elasticsearch.indices.recovery.RecoverySourceHandler$$Lambda$6706/0x0000000801a21480.accept(Unknown Source)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$recoverFilesFromSourceAndSnapshot$32(RecoverySourceHandler.java:762)
at org.elasticsearch.indices.recovery.RecoverySourceHandler$$Lambda$6735/0x0000000801890000.accept(Unknown Source)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29)
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:186)
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:143)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471)
at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:340)
at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:328)
at org.elasticsearch.transport.InboundHandler$$Lambda$5302/0x000000080188aa90.run(Unknown Source)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:833)

Do we really need to block in this method, or can these actions just be fire-and-forget things? I couldn't see an obvious reason for needing to wait for them to complete.

(FWIW this was clearly caused by setting cluster.routing.allocation.node_concurrent_recoveries too high, but really we shouldn't deadlock in any configuration)

Relates #77466

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[original-brownbear]

I'll see if I can find a quick solution here, would be nice to have this fixed for benchmarks.