[CI] Various tests in ShrinkIndexIT fail with "expected at least one master-eligible node left"

[droberts195]

This failure has occurred many times in the last 36 hours: https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:relative,to:now))&_a=(columns:!(test,build-id,build_url,branch),filters:!(),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'class:%20%22org.elasticsearch.action.admin.indices.create.ShrinkIndexIT%22'),sort:!(time,desc))

Often the failing test is ShrinkIndexIT.testCreateShrinkWithIndexSort, see for example https://gradle.com/s/fglguuelmqobe

More recently a failure of ShrinkIndexIT.testShrinkCommitsMergeOnIdle was seen but with the same error message: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/2405/testReport/junit/org.elasticsearch.action.admin.indices.create/ShrinkIndexIT/testShrinkCommitsMergeOnIdle/

In both cases the error message is something like:

java.lang.AssertionError: expected at least one master-eligible node left in {node_sc1=org.elasticsearch.test.InternalTestCluster$NodeAndClient@2de106c0}

A REPRO command for 7.x is:

./gradlew :server:integTest --tests "org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testCreateShrinkWithIndexSort" -Dtests.seed=50AF68DFDA0BACBE -Dtests.security.manager=true -Dtests.locale=zh-TW -Dtests.timezone=America/North_Dakota/Center -Dcompiler.java=12 -Druntime.java=8

A REPRO command for master is:

./gradlew :server:integTest --tests "org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkCommitsMergeOnIdle" -Dtests.seed=C57BD4CF13C2F399 -Dtests.security.manager=true -Dtests.locale=fo -Dtests.timezone=Europe/Zagreb -Dcompiler.java=12 -Druntime.java=11

Neither of these reproduce locally for me.

I will mute the suite.

[elasticmachine]

Pinging @elastic/es-distributed

[droberts195]

Muted on master in 2f4905f and on 7.x in cad804d

[original-brownbear]

I can easily reproduce this on master using seed -Dtests.seed=C57BD4CF13C2F399 and running the test suite on repeat in Idea (for by class). Trying to track this down now

[bizybot]

Muted on 7.3 in 52e72db
as it failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+multijob-unix-compatibility/os=amazon/17/testReport/junit/org.elasticsearch.action.admin.indices.create/ShrinkIndexIT/testCreateShrinkIndex/