Add option to take currently relocating shards' sizes into account #7785
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Because running out of disk space is such a hard failure condition I'd suggest that this should be the default behavior. Temporarily over-estimating seems safer to me. Great improvement though. Thanks! |
|
I also favor safety over accuracy by default. @dakrone what would the example situation that you mention ultimately resolve to? Would the same shards still be allocated, but slower than usual? |
| @@ -649,6 +654,107 @@ public void freeDiskPercentageAfterShardAssignedUnitTest() { | |||
| assertThat(after, equalTo(19.0)); | |||
| } | |||
|
|
|||
| @Test | |||
| @TestLogging("cluster.routing.allocation.decider:TRACE") | |||
There was a problem hiding this comment.
is trace logging required here?
|
I left a minor comment. I think we should move to |
dakrone
force-pushed
the
dtd-user-current-relocations
branch
2 times, most recently
from
be53e08 to
4185566
Compare
When using the DiskThresholdDecider, it's possible that shards could already be marked as relocating to the node being evaluated. This commit adds a new setting `cluster.routing.allocation.disk.include_relocations` which adds the size of the shards currently being relocated to this node to the node's used disk space. This new option defaults to `true`, however it's possible to over-estimate the usage for a node if the relocation is already partially complete, for instance: A node with a 10gb shard that's 45% of the way through a relocation would add 10gb + (.45 * 10) = 14.5gb to the node's disk usage before examining the watermarks to see if a new shard can be allocated. Fixes elastic#7753 Relates to elastic#6168
lcawl
added
:Distributed Indexing/Distributed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using the DiskThresholdDecider, it's possible that shards could
already be marked as relocating to the node being evaluated. This commit
adds a new setting
cluster.routing.allocation.disk.include_relocationswhich adds the size of the shards currently being relocated to this node
to the node's used disk space.
This new option defaults to
true, however, it's possible toover-estimate the usage for a node if the relocation is already
partially complete, for instance:
A node with a 10gb shard that's 45% of the way through a relocation
would add 10gb + (.45 * 10) = 14.5gb to the node's disk usage before
examining the watermarks to see if a new shard can be allocated.
Fixes #7753
Relates to #6168