Introduce separate shard limit for frozen shards #71392
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Frozen indices (partial searchable snapshots) require less heap per shard and the limit can therefore be raised for those. We pick 3000 frozen shards per frozen data node, since we think 2000 is reasonable to use in production. Relates elastic#71042
henningandersen
added
>enhancement
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
Pinging @elastic/es-core-features (Team:Core/Features) |
jasontedor
approved these changes
Apr 13, 2021
| private Optional<String> checkShardLimit(int newShards, int newFrozenShards, ClusterState state) { | ||
| // we verify the two limits independently. This also means that if they have mixed frozen and other data-roles nodes, such a mixed | ||
| // node can have both 1000 normal and 3000 frozen shards. This is the trade-off to keep the simplicity of the counts. We advocate | ||
| // against such mixed nodes for production use anyway. |
There was a problem hiding this comment.
I agree with this, good comment.
|
|
||
| private static boolean hasFrozen(DiscoveryNode node) { | ||
| final List<DiscoveryNodeRole> dataRoles = | ||
| node.getRoles().stream().filter(DiscoveryNodeRole::canContainData).collect(Collectors.toUnmodifiableList()); |
There was a problem hiding this comment.
How about return node.getRoles().contains(DiscoveryNodeRole.DATA_FROZEN_NODE_ROLE);? (And sorry if that doesn’t quite compile, I’m reviewing from my iPad, so no IDE support.)
There was a problem hiding this comment.
Thanks, much nicer and did compile, changed in 67e92fd
|
|
||
| private static boolean hasNonFrozen(DiscoveryNode node) { | ||
| final List<DiscoveryNodeRole> dataRoles = | ||
| node.getRoles().stream().filter(DiscoveryNodeRole::canContainData).collect(Collectors.toUnmodifiableList()); |
There was a problem hiding this comment.
return node.getRoles().stream().anyMatch(r -> r.canContainData() && r !=DiscoveryNodeRole.DATA_FROZEN_NODE_ROLE);?
There was a problem hiding this comment.
This suggested predicate almost exactly matches the name of the method (it has a non-frozen data role), while the current implementation requires a little bit of though. And we can avoid the unnecessary materialization of a collection, etc.
There was a problem hiding this comment.
Thanks, that is nicer, added in 67e92fd
|
@elasticmachine update branch |
|
@elasticmachine update branch |
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Apr 15, 2021
Frozen indices (partial searchable snapshots) require less heap per shard and the limit can therefore be raised for those. We pick 3000 frozen shards per frozen data node, since we think 2000 is reasonable to use in production. Relates elastic#71042 and elastic#34021
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Apr 16, 2021
Frozen indices created on 7.12 would not belong to the frozen shard limit group, now we convert them when last node is upgraded. Relates elastic#71392
henningandersen
added a commit
that referenced
this pull request
Apr 17, 2021
If master ends up on a newer version than other cluster members, we cannot apply the new index setting for shard limits. We skip doing so for now. Relates #71392
henningandersen
added a commit
that referenced
this pull request
Apr 20, 2021
Frozen indices created on 7.12 would not belong to the frozen shard limit group, now we convert them when last node is upgraded. Relates #71392
henningandersen
added a commit
that referenced
this pull request
Apr 20, 2021
Frozen indices created on 7.12 would not belong to the frozen shard limit group, now we convert them when last node is upgraded. Relates #71392
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Frozen indices (partial searchable snapshots) require less heap per
shard and the limit can therefore be raised for those. We pick 3000
frozen shards per frozen data node, since we think 2000 is reasonable
to use in production.
Relates #71042 and #34021