Defer reroute when nodes join #42855
Conversation
Today the master eagerly reroutes the cluster as part of processing node joins. However, it is not necessary to do this reroute straight away, and it is sometimes preferable to defer it until later. For instance, when the master wins its election it processes joins and performs a reroute, but it would be better to defer the reroute until after the master has become properly established. This change defers this reroute into a separate task, and batches multiple such tasks together.
DaveCTurner
added
>enhancement
:Distributed Coordination/Cluster Coordination
|
Pinging @elastic/es-distributed |
|
@ywelsch I managed to synthesise a failure by changing some of the |
| @@ -141,15 +142,17 @@ | |||
| public Coordinator(String nodeName, Settings settings, ClusterSettings clusterSettings, TransportService transportService, | |||
There was a problem hiding this comment.
I think we're at the point when the number of Coordinator constructor parameters are unmanageable. Can we possibly add a JavaDoc to the constructor describing how specific dependency is used by Coordinator?
There was a problem hiding this comment.
I know what you mean, but most of them are of very specific types. I added docs for the ones whose types don't make their meaning so clear in d9dbf2d (including the one added here).
| equalTo(false)); | ||
|
|
||
| final AtomicBoolean stopRerouting = new AtomicBoolean(); | ||
| final Thread rerouteThread = new Thread(() -> { |
There was a problem hiding this comment.
I'm not sure I understand what reroute thread is doing here. I mean I understand that it continuously performs reroutes, but why is it needed in the test? Can you add the comment, please?
There was a problem hiding this comment.
I found a simpler way to test the same thing now that I understand what's going on a bit better. See 563ea02.
| @@ -1243,7 +1243,8 @@ public void start(ClusterState initialState) { | |||
| allocationService, masterService, () -> persistedState, | |||
| hostsResolver -> testClusterNodes.nodes.values().stream().filter(n -> n.node.isMasterNode()) | |||
| .map(n -> n.node.getAddress()).collect(Collectors.toList()), | |||
| clusterService.getClusterApplierService(), Collections.emptyList(), random()); | |||
| clusterService.getClusterApplierService(), Collections.emptyList(), random(), | |||
| s -> {}); | |||
There was a problem hiding this comment.
perhaps plug in in the actual RoutingService here. This test cares about shards and realistic mocking, so I'm worried that a lot of effort will be spend in the future here to figure out why shards are not allocated.
Today the master eagerly reroutes the cluster as part of processing node joins. However, it is not necessary to do this reroute straight away, and it is sometimes preferable to defer it until later. For instance, when the master wins its election it processes joins and performs a reroute, but it would be better to defer the reroute until after the master has become properly established. This change defers this reroute into a separate task, and batches multiple such tasks together.
Today the master eagerly reroutes the cluster as part of processing node joins.
However, it is not necessary to do this reroute straight away, and it is
sometimes preferable to defer it until later. For instance, when the master
wins its election it processes joins and performs a reroute, but it would be
better to defer the reroute until after the master has become properly
established.
This change defers this reroute into a separate task.