Only connect to new nodes on new cluster state #39629
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today, when applying new cluster state we attempt to connect to all of its nodes as a blocking part of the application process. This is the right thing to do with new nodes, and is a no-op on any already-connected nodes, but is questionable on known nodes from which we are currently disconnected: there is a risk that we are partitioned from these nodes so that any attempt to connect to them will hang until it times out. This can dramatically slow down the application of new cluster states which hinders the recovery of the cluster during certain kinds of partition. If nodes are disconnected from the master then it is likely that they are to be removed as part of a subsequent cluster state update, so there's no need to try and reconnect to them like this. Moreover there is no need to attempt to reconnect to disconnected nodes as part of the cluster state application process, because we periodically try and reconnect to any disconnected nodes, and handle their disconnectedness reasonably gracefully in the meantime. This commit alters this behaviour to avoid reconnecting to known nodes during cluster state application. Resolves elastic#29025. Supersedes elastic#31547.
DaveCTurner
added
>enhancement
:Distributed Coordination/Network
|
Pinging @elastic/es-distributed |
DaveCTurner
added
:Distributed Coordination/Cluster Coordination
andrershov
reviewed
Mar 4, 2019
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Outdated
Show resolved
Hide resolved
| "connection cancelled by disconnection"); | ||
| } | ||
|
|
||
| Runnable ensureConnected(ActionListener<Void> listener) { |
There was a problem hiding this comment.
Is it possible that ensureConnected and connect/disconnect are called in different threads at the same time? I'm not sure how we're protecting listeners from races
There was a problem hiding this comment.
Yes, they can be called in different threads, but we only ever read or write listeners under the mutex. The listeners are never called under the mutex so it is possible that the notifications happen out of order, but this is benign.
|
@andrershov you'll be pleased to hear I adjusted this to use a future 😁 |
There was a problem hiding this comment.
Thanks @DaveCTurner , I have left some comments, otherwise looking good.
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Outdated
Show resolved
Hide resolved
henningandersen
approved these changes
Mar 12, 2019
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/cluster/NodeConnectionsServiceTests.java
Outdated
Show resolved
Hide resolved
andrershov
approved these changes
Mar 12, 2019
server/src/main/java/org/elasticsearch/cluster/NodeConnectionsService.java
Show resolved
Hide resolved
DaveCTurner
deleted the
2019-03-02-nodeconnectionsservice-avoid-blocking-on-known-nodes
branch
DaveCTurner
added a commit
that referenced
this pull request
Mar 12, 2019
Today, when applying new cluster state we attempt to connect to all of its nodes as a blocking part of the application process. This is the right thing to do with new nodes, and is a no-op on any already-connected nodes, but is questionable on known nodes from which we are currently disconnected: there is a risk that we are partitioned from these nodes so that any attempt to connect to them will hang until it times out. This can dramatically slow down the application of new cluster states which hinders the recovery of the cluster during certain kinds of partition. If nodes are disconnected from the master then it is likely that they are to be removed as part of a subsequent cluster state update, so there's no need to try and reconnect to them like this. Moreover there is no need to attempt to reconnect to disconnected nodes as part of the cluster state application process, because we periodically try and reconnect to any disconnected nodes, and handle their disconnectedness reasonably gracefully in the meantime. This commit alters this behaviour to avoid reconnecting to known nodes during cluster state application. Resolves #29025.
This was referenced Mar 13, 2019
colings86
added
>enhancement
and removed
:Distributed Coordination/Network
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, when applying new cluster state we attempt to connect to all of its
nodes as a blocking part of the application process. This is the right thing to
do with new nodes, and is a no-op on any already-connected nodes, but is
questionable on known nodes from which we are currently disconnected: there is
a risk that we are partitioned from these nodes so that any attempt to connect
to them will hang until it times out. This can dramatically slow down the
application of new cluster states which hinders the recovery of the cluster
during certain kinds of partition.
If nodes are disconnected from the master then it is likely that they are to be
removed as part of a subsequent cluster state update, so there's no need to try
and reconnect to them like this. Moreover there is no need to attempt to
reconnect to disconnected nodes as part of the cluster state application
process, because we periodically try and reconnect to any disconnected nodes,
and handle their disconnectedness reasonably gracefully in the meantime.
This commit alters this behaviour to avoid reconnecting to known nodes during
cluster state application.
Resolves #29025.
Supersedes #31547.