Skip zone/host awareness with auto-expand replicas #69334
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today if an index is set to `auto_expand_replicas: N-all` then we will try and create a shard copy on every node that matches the applicable allocation filters. This conflits with shard allocation awareness and the same-host allocation decider if there is an uneven distribution of nodes across zones or hosts, since these deciders prevent shard copies from being allocated unevenly and may therefore leave some unassigned shards. The point of these two deciders is to improve resilience given a limited number of shard copies but there is no need for this behaviour when the number of shard copies is not limited, so this commit supresses them in that case. Closes elastic#54151 Closes elastic#2869
DaveCTurner
added
>enhancement
:Distributed Coordination/Allocation
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
I'm saying that this closes #2869 because I could not see anything in the comments or issues linked to that issue that would still be worth addressing after this change. In general I don't see a use for setting |
ywelsch
approved these changes
Feb 22, 2021
| and <<allocation-total-shards,total shards per node>>, and this can lead to the | ||
| cluster health becoming `YELLOW` if the applicable rules prevent all the replicas | ||
| from being allocated. | ||
| Auto-expand the number of replicas based on the number of data nodes in the |
There was a problem hiding this comment.
Why remove the indentation? Does this still display properly?
There was a problem hiding this comment.
I don't know if it's possible to add paragraph breaks (the + on its own line) to one of these blocks of text otherwise.
DaveCTurner
deleted the
2021-02-22-auto-expand-replicas-vs-allocation-awareness
branch
DaveCTurner
added a commit
that referenced
this pull request
Feb 22, 2021
Today if an index is set to `auto_expand_replicas: N-all` then we will try and create a shard copy on every node that matches the applicable allocation filters. This conflits with shard allocation awareness and the same-host allocation decider if there is an uneven distribution of nodes across zones or hosts, since these deciders prevent shard copies from being allocated unevenly and may therefore leave some unassigned shards. The point of these two deciders is to improve resilience given a limited number of shard copies but there is no need for this behaviour when the number of shard copies is not limited, so this commit supresses them in that case. Closes #54151 Closes #2869
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Feb 22, 2021
Today we count `null` (i.e. missing) as a valid attribute value in allocation awareness, even though allocation awareness forbids the allocation of shards to such a node. Prior to elastic#69334 this didn't matter, a data node without allocation attributes was pointless. However, elastic#69334 means we now can allocate shards to such a node: for instance, there is no need for nodes holding only enrich indices to have allocation attributes. Therefore we should stop counting `null` as one of the attribute values.
DaveCTurner
added a commit
that referenced
this pull request
Feb 23, 2021
Today we count `null` (i.e. missing) as a valid attribute value in allocation awareness, even though allocation awareness forbids the allocation of shards to such a node. Prior to #69334 this didn't matter, a data node without allocation attributes was pointless. However, #69334 means we now can allocate shards to such a node: for instance, there is no need for nodes holding only enrich indices to have allocation attributes. Therefore we should stop counting `null` as one of the attribute values.
DaveCTurner
added a commit
that referenced
this pull request
Feb 23, 2021
Today we count `null` (i.e. missing) as a valid attribute value in allocation awareness, even though allocation awareness forbids the allocation of shards to such a node. Prior to #69334 this didn't matter, a data node without allocation attributes was pointless. However, #69334 means we now can allocate shards to such a node: for instance, there is no need for nodes holding only enrich indices to have allocation attributes. Therefore we should stop counting `null` as one of the attribute values.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today if an index is set to
auto_expand_replicas: N-allthen we willtry and create a shard copy on every node that matches the applicable
allocation filters. This conflits with shard allocation awareness and
the same-host allocation decider if there is an uneven distribution of
nodes across zones or hosts, since these deciders prevent shard copies
from being allocated unevenly and may therefore leave some unassigned
shards.
The point of these two deciders is to improve resilience given a limited
number of shard copies but there is no need for this behaviour when the
number of shard copies is not limited, so this commit supresses them in
that case.
Closes #54151
Closes #2869