[CI] TranslogTests#testFatalIOExceptionsWhileWritingConcurrently times out

[javanna]

TranslogTests#testFatalIOExceptionsWhileWritingConcurrently fails with a suite timeout error. This happened a couple of times a day in the last few days.

Example of a recent failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos/2300/console .

I can't reproduce this failure with the seed, but it is rather easy to reproduce it just by running this test some times (I tried once with iters set to 100 and it failed pretty quickly). Sounds like it's a timing issue then. I wonder if it may be caused by recent changes in #29401 that were fixing another failure for this same test method.

REPRODUCE WITH: ./gradlew :server:test \
  -Dtests.seed=9842CD3971C5E19D \
  -Dtests.class=org.elasticsearch.index.translog.TranslogTests \
  -Dtests.method="testFatalIOExceptionsWhileWritingConcurrently" \
  -Dtests.security.manager=true \
  -Dtests.locale=he-IL \
  -Dtests.timezone=America/North_Dakota/Beulah

REPRODUCE WITH: ./gradlew :server:test \
  -Dtests.seed=9842CD3971C5E19D \
  -Dtests.class=org.elasticsearch.index.translog.TranslogTests \
  -Dtests.security.manager=true \
  -Dtests.locale=he-IL \
  -Dtests.timezone=America/North_Dakota/Beulah

[javanna]

@jasontedor do you have a chance to have a look at this given that you touched this test a few days ago?

[elasticmachine]

Pinging @elastic/es-distributed

[javanna]

another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/6116/console .

[javanna]

Test muted in master and 6.x branches.

[jasontedor]

A translog thread can deadlock itself if it holds the read lock (Translog#readOperation) and needs to close the translog on a tragic event. When closing, we try to acquire a write lock. Since the thread already holds a read lock and this can not be upgraded to a write lock, it deadlocks waiting for itself to release the read lock.

[jasontedor]

I opened #29520.