From d0e6946c53e0dd3cf382cb9574ef6f7ff96c802a Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 1 Dec 2020 10:33:56 +0000 Subject: [PATCH 1/7] Expand docs on disk-based shard allocation Today we document the settings used to control rebalancing and disk-based shard allocation but there isn't really any discussion around what these processes do so it's hard to know what, if any, adjustments to make. This commit adds some words to help folk understand this area better. --- .../modules/cluster/disk_allocator.asciidoc | 57 ++++++++++++++++++- .../cluster/shards_allocation.asciidoc | 19 ++++++- 2 files changed, 70 insertions(+), 6 deletions(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index c16109e901c4d..78cddc09cff89 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -2,9 +2,60 @@ ==== Disk-based shard allocation settings [[disk-based-shard-allocation-description]] // tag::disk-based-shard-allocation-description-tag[] -{es} considers the available disk space on a node before deciding -whether to allocate new shards to that node or to actively relocate shards away -from that node. + +The <> of the cluster depends only on the +number of shards on each node and the indices to which those shards belong. It +considers neither the sizes of these shards nor the available disk space on +each node, for the following reasons: + +* Disk usage changes over time. Balancing the disk usage of individual nodes +would require a lot more shard movements, perhaps even wastefully undoing +earlier movements. Moving a shard consumes resources such as I/O and network +bandwidth and may evict data from the filesystem cache. These resources are +better spent handling your searches and indexing where possible. + +* A cluster with accurately-balanced disk usage typically performs no better +than one that has unequal disk usage across its nodes, as long as no disk is +too full. + +The disk-based shard allocator ensures that all nodes have enough disk space +without performing more shard movements than necessary. It allocates shards +based on a pair of thresholds known as the _low watermark_ and the _high +watermark_. Its primary goal is to ensure that no node breaches the high +watermark, or at least that any such breach is only temporary. If a node +breaches the high watermark then {es} will solve this by moving some of its +shards onto other nodes in the cluster. + +NOTE: It is normal for nodes to temporarily exceed the high watermark from time +to time. + +The allocator also tries to keep nodes clear of the high watermark by +forbidding the allocation of more shards on a node that exceeds the low +watermark. Importantly, if all of your nodes have exceeded the low watermark +then no new shards can be allocated and {es} will not be able to move any +shards between nodes in order to keep the disk usage below the high watermark. +You must ensure that your cluster has enough disk space in total and that that +there are always some nodes that are below the low watermark. + +Shard movements triggered by the disk-based shard allocator must also satisfy +all other shard allocation rules such as +<> and +<>. If these rules are too strict then they +can also prevent the shard movements needed to keep the nodes' disk usage under +control. + +If a node is filling up its disk faster than {es} can move shards elsewhere +then there is a risk that the disk will completely fill up. To prevent this, as +a last resort, once the disk usage reaches the _flood-stage_ watermark {es} +will block further writes to the indices which have a shard on the affected +node. It will also continue to move shards onto the other nodes in the cluster. +Once the disk usage on the affected node has dropped below the high watermark, +the write block will be removed automatically. + +[[disk-based-shard-allocation-does-not-balance]] +IMPORTANT: It is completely normal for the nodes in your cluster to be using +very different amounts of disk space. As long as any breaches of the high +watermark are only temporary, {es} is working as expected. You can use the following settings to control disk-based allocation: diff --git a/docs/reference/modules/cluster/shards_allocation.asciidoc b/docs/reference/modules/cluster/shards_allocation.asciidoc index f72450feb93a3..51d2b588f1928 100644 --- a/docs/reference/modules/cluster/shards_allocation.asciidoc +++ b/docs/reference/modules/cluster/shards_allocation.asciidoc @@ -97,9 +97,22 @@ Specify when shard rebalancing is allowed: [[shards-rebalancing-heuristics]] ==== Shard balancing heuristics settings -The following settings are used together to determine where to place each -shard. The cluster is balanced when no allowed rebalancing operation can bring the weight -of any node closer to the weight of any other node by more than the `balance.threshold`. +A cluster is _balanced_ when it has an equal number of shards on each node +without having a concentration of shards from any index on any node. {es} runs +an automatic process called _rebalancing_ which moves shards between the nodes +in your cluster in order to improve its balance. Rebalancing obeys all other +shard allocation rules such as <> and <> which may prevent it from +completely balancing the cluster. In that case, rebalancing strives to acheve +the most balanced cluster possible within the rules you have configured. + +Rebalancing works by computing a _weight_ for each node based on its allocation +of shards, and then moving shards between nodes to reduce the weight of the +heavier nodes and increase the weight of the lighter ones. The cluster is +balanced when there is no possible shard movement that can bring the weight of +any node closer to the weight of any other node by more than a configurable +threshold. The following settings allow you to control the details of these +calculations. `cluster.routing.allocation.balance.shard`:: (<>) From 8985fa1e32e8afd00e030a4e65a80020a47f44e2 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 1 Dec 2020 11:23:48 +0000 Subject: [PATCH 2/7] Every damn time --- docs/reference/modules/cluster/disk_allocator.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index 78cddc09cff89..2b6a9bc578edb 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -3,7 +3,7 @@ [[disk-based-shard-allocation-description]] // tag::disk-based-shard-allocation-description-tag[] -The <> of the cluster depends only on the +The <> of the cluster depends only on the number of shards on each node and the indices to which those shards belong. It considers neither the sizes of these shards nor the available disk space on each node, for the following reasons: From 67a9e61585dce5833a2ce0d378022fe740d310bf Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 7 Dec 2020 13:22:38 +0000 Subject: [PATCH 3/7] Apply suggestions from code review Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> --- .../modules/cluster/disk_allocator.asciidoc | 22 +++++++++---------- .../cluster/shards_allocation.asciidoc | 2 +- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index 2b6a9bc578edb..d87a2a514a319 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -21,9 +21,9 @@ too full. The disk-based shard allocator ensures that all nodes have enough disk space without performing more shard movements than necessary. It allocates shards based on a pair of thresholds known as the _low watermark_ and the _high -watermark_. Its primary goal is to ensure that no node breaches the high -watermark, or at least that any such breach is only temporary. If a node -breaches the high watermark then {es} will solve this by moving some of its +watermark_. Its primary goal is to ensure that no node exceeds the high +watermark, or at least that any such overage is only temporary. If a node +exceeds the high watermark then {es} will solve this by moving some of its shards onto other nodes in the cluster. NOTE: It is normal for nodes to temporarily exceed the high watermark from time @@ -34,8 +34,8 @@ forbidding the allocation of more shards on a node that exceeds the low watermark. Importantly, if all of your nodes have exceeded the low watermark then no new shards can be allocated and {es} will not be able to move any shards between nodes in order to keep the disk usage below the high watermark. -You must ensure that your cluster has enough disk space in total and that that -there are always some nodes that are below the low watermark. +You must ensure that your cluster has enough disk space in total and that +there are always some nodes below the low watermark. Shard movements triggered by the disk-based shard allocator must also satisfy all other shard allocation rules such as @@ -47,15 +47,15 @@ control. If a node is filling up its disk faster than {es} can move shards elsewhere then there is a risk that the disk will completely fill up. To prevent this, as a last resort, once the disk usage reaches the _flood-stage_ watermark {es} -will block further writes to the indices which have a shard on the affected +will block writes to indices with a shard on the affected node. It will also continue to move shards onto the other nodes in the cluster. -Once the disk usage on the affected node has dropped below the high watermark, -the write block will be removed automatically. +When disk usage on the affected node drops below the high watermark, +{es} automatically removes the write block. [[disk-based-shard-allocation-does-not-balance]] -IMPORTANT: It is completely normal for the nodes in your cluster to be using -very different amounts of disk space. As long as any breaches of the high -watermark are only temporary, {es} is working as expected. +IMPORTANT: It is normal for a cluster's nodes to use different amounts of disk +space. As long as any high watermark overages are temporary, {es} is +working as expected. You can use the following settings to control disk-based allocation: diff --git a/docs/reference/modules/cluster/shards_allocation.asciidoc b/docs/reference/modules/cluster/shards_allocation.asciidoc index 51d2b588f1928..54ded051d5a5f 100644 --- a/docs/reference/modules/cluster/shards_allocation.asciidoc +++ b/docs/reference/modules/cluster/shards_allocation.asciidoc @@ -100,7 +100,7 @@ Specify when shard rebalancing is allowed: A cluster is _balanced_ when it has an equal number of shards on each node without having a concentration of shards from any index on any node. {es} runs an automatic process called _rebalancing_ which moves shards between the nodes -in your cluster in order to improve its balance. Rebalancing obeys all other +in your cluster to improve its balance. Rebalancing obeys all other shard allocation rules such as <> and <> which may prevent it from completely balancing the cluster. In that case, rebalancing strives to acheve From b21fa1e7d001594fed8286633d25e70f72ae48ff Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 7 Dec 2020 13:44:47 +0000 Subject: [PATCH 4/7] Move balance-based intro lower down --- .../modules/cluster/disk_allocator.asciidoc | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index d87a2a514a319..dcc4ac654b28f 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -3,21 +3,6 @@ [[disk-based-shard-allocation-description]] // tag::disk-based-shard-allocation-description-tag[] -The <> of the cluster depends only on the -number of shards on each node and the indices to which those shards belong. It -considers neither the sizes of these shards nor the available disk space on -each node, for the following reasons: - -* Disk usage changes over time. Balancing the disk usage of individual nodes -would require a lot more shard movements, perhaps even wastefully undoing -earlier movements. Moving a shard consumes resources such as I/O and network -bandwidth and may evict data from the filesystem cache. These resources are -better spent handling your searches and indexing where possible. - -* A cluster with accurately-balanced disk usage typically performs no better -than one that has unequal disk usage across its nodes, as long as no disk is -too full. - The disk-based shard allocator ensures that all nodes have enough disk space without performing more shard movements than necessary. It allocates shards based on a pair of thresholds known as the _low watermark_ and the _high @@ -53,9 +38,23 @@ When disk usage on the affected node drops below the high watermark, {es} automatically removes the write block. [[disk-based-shard-allocation-does-not-balance]] -IMPORTANT: It is normal for a cluster's nodes to use different amounts of disk -space. As long as any high watermark overages are temporary, {es} is -working as expected. +[TIP] +==== +It is normal for the nodes in your cluster to be using very +different amounts of disk space. The <> +of the cluster depends only on the number of shards on each node and the +indices to which those shards belong. It considers neither the sizes of these +shards nor the available disk space on each node, for the following reasons: + +* Disk usage changes over time. Balancing the disk usage of individual nodes +would require a lot more shard movements, perhaps even wastefully undoing +earlier movements. Moving a shard consumes resources such as I/O and network +bandwidth and may evict data from the filesystem cache. These resources are +better spent handling your searches and indexing where possible. + +* A cluster with equal disk usage on every node typically performs no better +than one that has unequal disk usage, as long as no disk is too full. +==== You can use the following settings to control disk-based allocation: From c3c38e493e3aae83ac894622f07fcb8b8b864c9b Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 7 Dec 2020 13:50:02 +0000 Subject: [PATCH 5/7] Mention data tiers w.r.t. disk allocation and balancing --- docs/reference/modules/cluster/disk_allocator.asciidoc | 5 ++++- .../reference/modules/cluster/shards_allocation.asciidoc | 9 ++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index dcc4ac654b28f..5ada710c3b153 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -27,7 +27,10 @@ all other shard allocation rules such as <> and <>. If these rules are too strict then they can also prevent the shard movements needed to keep the nodes' disk usage under -control. +control. If you are using <> then {es} automatically +configures allocation filtering rules to place shards within the appropriate +tier, which means that the disk-based shard allocator works independently +within each tier. If a node is filling up its disk faster than {es} can move shards elsewhere then there is a risk that the disk will completely fill up. To prevent this, as diff --git a/docs/reference/modules/cluster/shards_allocation.asciidoc b/docs/reference/modules/cluster/shards_allocation.asciidoc index 54ded051d5a5f..fe316c988b344 100644 --- a/docs/reference/modules/cluster/shards_allocation.asciidoc +++ b/docs/reference/modules/cluster/shards_allocation.asciidoc @@ -100,11 +100,14 @@ Specify when shard rebalancing is allowed: A cluster is _balanced_ when it has an equal number of shards on each node without having a concentration of shards from any index on any node. {es} runs an automatic process called _rebalancing_ which moves shards between the nodes -in your cluster to improve its balance. Rebalancing obeys all other -shard allocation rules such as <> and <> which may prevent it from completely balancing the cluster. In that case, rebalancing strives to acheve -the most balanced cluster possible within the rules you have configured. +the most balanced cluster possible within the rules you have configured. If you +are using <> then {es} automatically applies allocation +filtering rules to place each shard within the appropriate tier. These rules +mean that the balancer works independently within each tier. Rebalancing works by computing a _weight_ for each node based on its allocation of shards, and then moving shards between nodes to reduce the weight of the From c59d8ae8237edce23a86e2964154e94f202045cf Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 7 Dec 2020 14:30:20 +0000 Subject: [PATCH 6/7] Better location for balancing intro --- .../cluster/shards_allocation.asciidoc | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/docs/reference/modules/cluster/shards_allocation.asciidoc b/docs/reference/modules/cluster/shards_allocation.asciidoc index fe316c988b344..e26e732e3455e 100644 --- a/docs/reference/modules/cluster/shards_allocation.asciidoc +++ b/docs/reference/modules/cluster/shards_allocation.asciidoc @@ -56,10 +56,21 @@ one of the active allocation ids in the cluster state. [[shards-rebalancing-settings]] ==== Shard rebalancing settings +A cluster is _balanced_ when it has an equal number of shards on each node +without having a concentration of shards from any index on any node. {es} runs +an automatic process called _rebalancing_ which moves shards between the nodes +in your cluster to improve its balance. Rebalancing obeys all other shard +allocation rules such as <> and <> which may prevent it from +completely balancing the cluster. In that case, rebalancing strives to acheve +the most balanced cluster possible within the rules you have configured. If you +are using <> then {es} automatically applies allocation +filtering rules to place each shard within the appropriate tier. These rules +mean that the balancer works independently within each tier. + You can use the following settings to control the rebalancing of shards across the cluster: - `cluster.routing.rebalance.enable`:: + -- @@ -97,18 +108,6 @@ Specify when shard rebalancing is allowed: [[shards-rebalancing-heuristics]] ==== Shard balancing heuristics settings -A cluster is _balanced_ when it has an equal number of shards on each node -without having a concentration of shards from any index on any node. {es} runs -an automatic process called _rebalancing_ which moves shards between the nodes -in your cluster to improve its balance. Rebalancing obeys all other shard -allocation rules such as <> and <> which may prevent it from -completely balancing the cluster. In that case, rebalancing strives to acheve -the most balanced cluster possible within the rules you have configured. If you -are using <> then {es} automatically applies allocation -filtering rules to place each shard within the appropriate tier. These rules -mean that the balancer works independently within each tier. - Rebalancing works by computing a _weight_ for each node based on its allocation of shards, and then moving shards between nodes to reduce the weight of the heavier nodes and increase the weight of the lighter ones. The cluster is From 331e9b6b386f8cb133ad07c74bfb1754152afa76 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 7 Dec 2020 14:31:18 +0000 Subject: [PATCH 7/7] on -> to; reformat --- .../modules/cluster/disk_allocator.asciidoc | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index 5ada710c3b153..fc6d9f15eb09c 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -15,12 +15,12 @@ NOTE: It is normal for nodes to temporarily exceed the high watermark from time to time. The allocator also tries to keep nodes clear of the high watermark by -forbidding the allocation of more shards on a node that exceeds the low +forbidding the allocation of more shards to a node that exceeds the low watermark. Importantly, if all of your nodes have exceeded the low watermark then no new shards can be allocated and {es} will not be able to move any shards between nodes in order to keep the disk usage below the high watermark. -You must ensure that your cluster has enough disk space in total and that -there are always some nodes below the low watermark. +You must ensure that your cluster has enough disk space in total and that there +are always some nodes below the low watermark. Shard movements triggered by the disk-based shard allocator must also satisfy all other shard allocation rules such as @@ -35,19 +35,19 @@ within each tier. If a node is filling up its disk faster than {es} can move shards elsewhere then there is a risk that the disk will completely fill up. To prevent this, as a last resort, once the disk usage reaches the _flood-stage_ watermark {es} -will block writes to indices with a shard on the affected -node. It will also continue to move shards onto the other nodes in the cluster. -When disk usage on the affected node drops below the high watermark, -{es} automatically removes the write block. +will block writes to indices with a shard on the affected node. It will also +continue to move shards onto the other nodes in the cluster. When disk usage +on the affected node drops below the high watermark, {es} automatically removes +the write block. [[disk-based-shard-allocation-does-not-balance]] [TIP] ==== -It is normal for the nodes in your cluster to be using very -different amounts of disk space. The <> -of the cluster depends only on the number of shards on each node and the -indices to which those shards belong. It considers neither the sizes of these -shards nor the available disk space on each node, for the following reasons: +It is normal for the nodes in your cluster to be using very different amounts +of disk space. The <> of the cluster +depends only on the number of shards on each node and the indices to which +those shards belong. It considers neither the sizes of these shards nor the +available disk space on each node, for the following reasons: * Disk usage changes over time. Balancing the disk usage of individual nodes would require a lot more shard movements, perhaps even wastefully undoing