Expand docs on disk-based shard allocation (#65668)

Today we document the settings used to control rebalancing and
disk-based shard allocation but there isn't really any discussion around
what these processes do so it's hard to know what, if any, adjustments
to make.

This commit adds some words to help folk understand this area better.

Author: DaveCTurner

docs/reference/modules/cluster/disk_allocator.asciidoc

@@ -2,9 +2,62 @@
 ==== Disk-based shard allocation settings
 [[disk-based-shard-allocation-description]]
 // tag::disk-based-shard-allocation-description-tag[]
-{es} considers the available disk space on a node before deciding
-whether to allocate new shards to that node or to actively relocate shards away
-from that node.
+
+The disk-based shard allocator ensures that all nodes have enough disk space
+without performing more shard movements than necessary. It allocates shards
+based on a pair of thresholds known as the _low watermark_ and the _high
+watermark_. Its primary goal is to ensure that no node exceeds the high
+watermark, or at least that any such overage is only temporary. If a node
+exceeds the high watermark then {es} will solve this by moving some of its
+shards onto other nodes in the cluster.
+
+NOTE: It is normal for nodes to temporarily exceed the high watermark from time
+to time.
+
+The allocator also tries to keep nodes clear of the high watermark by
+forbidding the allocation of more shards to a node that exceeds the low
+watermark. Importantly, if all of your nodes have exceeded the low watermark
+then no new shards can be allocated and {es} will not be able to move any
+shards between nodes in order to keep the disk usage below the high watermark.
+You must ensure that your cluster has enough disk space in total and that there
+are always some nodes below the low watermark.
+
+Shard movements triggered by the disk-based shard allocator must also satisfy
+all other shard allocation rules such as
+<<cluster-shard-allocation-filtering,allocation filtering>> and
+<<forced-awareness,forced awareness>>. If these rules are too strict then they
+can also prevent the shard movements needed to keep the nodes' disk usage under
+control. If you are using <<data-tiers,data tiers>> then {es} automatically
+configures allocation filtering rules to place shards within the appropriate
+tier, which means that the disk-based shard allocator works independently
+within each tier.
+
+If a node is filling up its disk faster than {es} can move shards elsewhere
+then there is a risk that the disk will completely fill up. To prevent this, as
+a last resort, once the disk usage reaches the _flood-stage_ watermark {es}
+will block writes to indices with a shard on the affected node. It will also
+continue to move shards onto the other nodes in the cluster.  When disk usage
+on the affected node drops below the high watermark, {es} automatically removes
+the write block.
+
+[[disk-based-shard-allocation-does-not-balance]]
+[TIP]
+====
+It is normal for the nodes in your cluster to be using very different amounts
+of disk space. The <<shards-rebalancing-settings,balance>> of the cluster
+depends only on the number of shards on each node and the indices to which
+those shards belong. It considers neither the sizes of these shards nor the
+available disk space on each node, for the following reasons:
+
+* Disk usage changes over time. Balancing the disk usage of individual nodes
+would require a lot more shard movements, perhaps even wastefully undoing
+earlier movements. Moving a shard consumes resources such as I/O and network
+bandwidth and may evict data from the filesystem cache. These resources are
+better spent handling your searches and indexing where possible.
+
+* A cluster with equal disk usage on every node typically performs no better
+than one that has unequal disk usage, as long as no disk is too full.
+====
 
 You can use the following settings to control disk-based allocation:
 

docs/reference/modules/cluster/shards_allocation.asciidoc

@@ -56,10 +56,21 @@ one of the active allocation ids in the cluster state.
 [[shards-rebalancing-settings]]
 ==== Shard rebalancing settings
 
+A cluster is _balanced_ when it has an equal number of shards on each node
+without having a concentration of shards from any index on any node. {es} runs
+an automatic process called _rebalancing_ which moves shards between the nodes
+in your cluster to improve its balance. Rebalancing obeys all other shard
+allocation rules such as <<cluster-shard-allocation-filtering,allocation
+filtering>> and <<forced-awareness,forced awareness>> which may prevent it from
+completely balancing the cluster. In that case, rebalancing strives to acheve
+the most balanced cluster possible within the rules you have configured. If you
+are using <<data-tiers,data tiers>> then {es} automatically applies allocation
+filtering rules to place each shard within the appropriate tier. These rules
+mean that the balancer works independently within each tier.
+
 You can use the following settings to control the rebalancing of shards across
 the cluster:
 
-
 `cluster.routing.rebalance.enable`::
 +
 --
@@ -97,9 +108,13 @@ Specify when shard rebalancing is allowed:
 [[shards-rebalancing-heuristics]]
 ==== Shard balancing heuristics settings
 
-The following settings are used together to determine where to place each
-shard.  The cluster is balanced when no allowed rebalancing operation can bring the weight
-of any node closer to the weight of any other node by more than the `balance.threshold`.
+Rebalancing works by computing a _weight_ for each node based on its allocation
+of shards, and then moving shards between nodes to reduce the weight of the
+heavier nodes and increase the weight of the lighter ones. The cluster is
+balanced when there is no possible shard movement that can bring the weight of
+any node closer to the weight of any other node by more than a configurable
+threshold. The following settings allow you to control the details of these
+calculations.
 
 `cluster.routing.allocation.balance.shard`::
      (<<dynamic-cluster-setting,Dynamic>>)