Fix CoordinatorTests.testUnresponsiveLeaderDetectedEventually

[fcofdez]

Take into account messy scenarios of 5 node clusters elections
where multiple nodes can trigger an election concurrently, meaning
that it takes longer to stabilize the cluster and elect a leader.

Fixes #63918

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[DaveCTurner]

LGTM

[DaveCTurner]

Hmm actually I think we need another defaultMillis(PUBLISH_TIMEOUT_SETTING) too -- the failing elections all try and publish to the unresponsive leader and must therefore wait for a timeout.

[fcofdez]

I'm not sure if we need the additional publish timeout as those publications go through as expected (since they have quorum) or fail due to term bumps. Am I missing something?

[DaveCTurner]

Yes, until the master is properly established each publication will go to the unresponsive node and therefore wait for the publish timeout before proceeding. That node is only removed from the cluster once the elections have settled down.

In the failing test, we blackhole node4 at 433835372ms, and then attempt the following publications:

[433867822ms][+    32450ms][node4][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=3, version=6, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433868232ms][+    32860ms][node3][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=5, version=6, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433897846ms][+    62474ms][node4][o.e.c.c.C.CoordinatorPublication          ] publication ended unsuccessfully: Publication{term=3, version=6}
[433898290ms][+    62918ms][node3][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=5, version=6}
[433898645ms][+    63273ms][node3][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=8, version=7, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433898735ms][+    63363ms][node3][o.e.c.c.C.CoordinatorPublication          ] publication ended unsuccessfully: Publication{term=8, version=7}
[433899213ms][+    63841ms][node1][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=10, version=8, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433929341ms][+    93969ms][node1][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=10, version=8}
[433929686ms][+    94314ms][node1][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=13, version=9, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433959731ms][+   124359ms][node1][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=13, version=9}
[433959731ms][+   124359ms][node1][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=13, version=10, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433989746ms][+   154374ms][node1][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=13, version=10}
[433989826ms][+   154454ms][node1][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=13, version=11, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433989975ms][+   154603ms][node1][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=13, version=11}
[433990058ms][+   154686ms][node1][o.e.c.c.C.CoordinatorPublication          ] publishing PublishRequest{term=13, version=12, state=cluster uuid: H5lUTwAAQACNnOGK_____w [committed: true]
[433990366ms][+   154994ms][node1][o.e.c.c.C.CoordinatorPublication          ] publication ended successfully: Publication{term=13, version=12}

(second column is times relative to the blackhole time)

Until version 11, all publications go to node4 and therefore take ≥30 seconds because of hitting the publish timeout. In particular, this includes the clashing publication of version 6 which we're wanting to account for here.

[fcofdez]

Thanks for the explanation @DaveCTurner ! I was missing that publications waits up until the timeout to succeed even if it got a Quorum. I've updated the PR.

[DaveCTurner]

LGTM