Skip to content

Commit c9e03e6

Browse files
debadairdebadair
authored
[DOCS] Reworked the shard allocation filtering info. (#36456)
* [DOCS] Reworked the shard allocation filtering info. Closes #36079 * Added multiple index allocation settings example back. * Removed extraneous space
1 parent c3a6d19 commit c9e03e6

File tree

6 files changed

+182
-176
lines changed

6 files changed

+182
-176
lines changed

‎docs/reference/index-modules/allocation/filtering.asciidoc

+59-48
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,54 @@
11
[[shard-allocation-filtering]]
2-
=== Shard Allocation Filtering
3-
4-
Shard allocation filtering allows you to specify which nodes are allowed
5-
to host the shards of a particular index.
6-
7-
NOTE: The per-index shard allocation filters explained below work in
8-
conjunction with the cluster-wide allocation filters explained in
9-
<<shards-allocation>>.
10-
11-
It is possible to assign arbitrary metadata attributes to each node at
12-
startup. For instance, nodes could be assigned a `rack` and a `size`
13-
attribute as follows:
14-
2+
=== Index-level shard allocation filtering
3+
4+
You can use shard allocation filters to control where {es} allocates shards of
5+
a particular index. These per-index filters are applied in conjunction with
6+
<<allocation-filtering, cluster-wide allocation filtering>> and
7+
<<allocation-awareness, allocation awareness>>.
8+
9+
Shard allocation filters can be based on custom node attributes or the built-in
10+
`_name`, `host_ip`, `publish_ip`, `_ip`, and `_host` attributes.
11+
<<index-lifecycle-management, Index lifecycle management>> uses filters based
12+
on custom node attributes to determine how to reallocate shards when moving
13+
between phases.
14+
15+
The `cluster.routing.allocation` settings are dynamic, enabling live indices to
16+
be moved from one set of nodes to another. Shards are only relocated if it is
17+
possible to do so without breaking another routing constraint, such as never
18+
allocating a primary and replica shard on the same node.
19+
20+
For example, you could use a custom node attribute to indicate a node's
21+
performance characteristics and use shard allocation filtering to route shards
22+
for a particular index to the most appropriate class of hardware.
23+
24+
[float]
25+
[[index-allocation-filters]]
26+
==== Enabling index-level shard allocation filtering
27+
28+
To filter based on a custom node attribute:
29+
30+
. Specify the filter characteristics with a custom node attribute in each
31+
node's `elasticsearch.yml` configuration file. For example, if you have `small`,
32+
`medium`, and `big` nodes, you could add a `size` attribute to filter based
33+
on node size.
34+
+
35+
[source,yaml]
36+
--------------------------------------------------------
37+
node.attr.size: medium
38+
--------------------------------------------------------
39+
+
40+
You can also set custom attributes when you start a node:
41+
+
1542
[source,sh]
16-
------------------------
17-
bin/elasticsearch -Enode.attr.rack=rack1 -Enode.attr.size=big <1>
18-
------------------------
19-
<1> These attribute settings can also be specified in the `elasticsearch.yml` config file.
20-
21-
These metadata attributes can be used with the
22-
`index.routing.allocation.*` settings to allocate an index to a particular
23-
group of nodes. For instance, we can move the index `test` to either `big` or
24-
`medium` nodes as follows:
25-
26-
43+
--------------------------------------------------------
44+
`./bin/elasticsearch -Enode.attr.size=medium
45+
--------------------------------------------------------
46+
47+
. Add a routing allocation filter to the index. The `index.routing.allocation`
48+
settings support three types of filters: `include`, `exclude`, and `require`.
49+
For example, to tell {es} to allocate shards from the `test` index to either
50+
`big` or `medium` nodes, use `index.routing.allocation.include`:
51+
+
2752
[source,js]
2853
------------------------
2954
PUT test/_settings
@@ -33,24 +58,11 @@ PUT test/_settings
3358
------------------------
3459
// CONSOLE
3560
// TEST[s/^/PUT test\n/]
36-
37-
Alternatively, we can move the index `test` away from the `small` nodes with
38-
an `exclude` rule:
39-
40-
[source,js]
41-
------------------------
42-
PUT test/_settings
43-
{
44-
"index.routing.allocation.exclude.size": "small"
45-
}
46-
------------------------
47-
// CONSOLE
48-
// TEST[s/^/PUT test\n/]
49-
50-
Multiple rules can be specified, in which case all conditions must be
51-
satisfied. For instance, we could move the index `test` to `big` nodes in
52-
`rack1` with the following:
53-
61+
+
62+
If you specify multiple filters, all conditions must be satisfied for shards to
63+
be relocated. For example, to move the `test` index to `big` nodes in `rack1`,
64+
you could specify:
65+
+
5466
[source,js]
5567
------------------------
5668
PUT test/_settings
@@ -62,10 +74,9 @@ PUT test/_settings
6274
// CONSOLE
6375
// TEST[s/^/PUT test\n/]
6476

65-
NOTE: If some conditions cannot be satisfied then shards will not be moved.
66-
67-
The following settings are _dynamic_, allowing live indices to be moved from
68-
one set of nodes to another:
77+
[float]
78+
[[index-allocation-settings]]
79+
==== Index allocation filter settings
6980

7081
`index.routing.allocation.include.{attribute}`::
7182

@@ -82,7 +93,7 @@ one set of nodes to another:
8293
Assign the index to a node whose `{attribute}` has _none_ of the
8394
comma-separated values.
8495

85-
These special attributes are also supported:
96+
The index allocation settings support the following built-in attributes:
8697

8798
[horizontal]
8899
`_name`:: Match nodes by node name
@@ -91,7 +102,7 @@ These special attributes are also supported:
91102
`_ip`:: Match either `_host_ip` or `_publish_ip`
92103
`_host`:: Match nodes by hostname
93104

94-
All attribute values can be specified with wildcards, eg:
105+
You can use wildcards when specifying attribute values, for example:
95106

96107
[source,js]
97108
------------------------

‎docs/reference/index-modules/allocation/total_shards.asciidoc

+1-4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[[allocation-total-shards]]
2-
=== Total Shards Per Node
2+
=== Total shards per node
33

44
The cluster-level shard allocator tries to spread the shards of a single index
55
across as many nodes as possible. However, depending on how many shards and
@@ -28,6 +28,3 @@ allocated.
2828
2929
Use with caution.
3030
=======================================
31-
32-
33-

‎docs/reference/modules/cluster/allocation_awareness.asciidoc

+84-88
Original file line numberDiff line numberDiff line change
@@ -1,114 +1,110 @@
11
[[allocation-awareness]]
2-
=== Shard Allocation Awareness
2+
=== Shard allocation awareness
3+
4+
You can use custom node attributes as _awareness attributes_ to enable {es}
5+
to take your physical hardware configuration into account when allocating shards.
6+
If {es} knows which nodes are on the same physical server, in the same rack, or
7+
in the same zone, it can distribute the primary shard and its replica shards to
8+
minimise the risk of losing all shard copies in the event of a failure.
9+
10+
When shard allocation awareness is enabled with the
11+
`cluster.routing.allocation.awareness.attributes` setting, shards are only
12+
allocated to nodes that have values set for the specified awareness
13+
attributes. If you use multiple awareness attributes, {es} considers
14+
each attribute separately when allocating shards.
15+
16+
The allocation awareness settings can be configured in
17+
`elasticsearch.yml` and updated dynamically with the
18+
<<cluster-update-settings,cluster-update-settings>> API.
319

4-
When running nodes on multiple VMs on the same physical server, on multiple
5-
racks, or across multiple zones or domains, it is more likely that two nodes on
6-
the same physical server, in the same rack, or in the same zone or domain will
7-
crash at the same time, rather than two unrelated nodes crashing
8-
simultaneously.
20+
{es} prefers using shards in the same location (with the same
21+
awareness attribute values) to process search or GET requests. Using local
22+
shards is usually faster than crossing rack or zone boundaries.
923

10-
If Elasticsearch is _aware_ of the physical configuration of your hardware, it
11-
can ensure that the primary shard and its replica shards are spread across
12-
different physical servers, racks, or zones, to minimise the risk of losing
13-
all shard copies at the same time.
24+
NOTE: The number of attribute values determines how many shard copies are
25+
allocated in each location. If the number of nodes in each location is
26+
unbalanced and there are a lot of replicas, replica shards might be left
27+
unassigned.
1428

15-
The shard allocation awareness settings allow you to tell Elasticsearch about
16-
your hardware configuration.
29+
[float]
30+
[[enabling-awareness]]
31+
==== Enabling shard allocation awareness
1732

18-
As an example, let's assume we have several racks. When we start a node, we
19-
can tell it which rack it is in by assigning it an arbitrary metadata
20-
attribute called `rack_id` -- we could use any attribute name. For example:
33+
To enable shard allocation awareness:
2134

35+
. Specify the location of each node with a custom node attribute. For example,
36+
if you want Elasticsearch to distribute shards across different racks, you might
37+
set an awareness attribute called `rack_id` in each node's `elasticsearch.yml`
38+
config file.
39+
+
40+
[source,yaml]
41+
--------------------------------------------------------
42+
node.attr.rack_id: rack_one
43+
--------------------------------------------------------
44+
+
45+
You can also set custom attributes when you start a node:
46+
+
2247
[source,sh]
23-
----------------------
24-
./bin/elasticsearch -Enode.attr.rack_id=rack_one <1>
25-
----------------------
26-
<1> This setting could also be specified in the `elasticsearch.yml` config file.
27-
28-
Now, we need to set up _shard allocation awareness_ by telling Elasticsearch
29-
which attributes to use. This can be configured in the `elasticsearch.yml`
30-
file on *all* master-eligible nodes, or it can be set (and changed) with the
31-
<<cluster-update-settings,cluster-update-settings>> API.
32-
33-
For our example, we'll set the value in the config file:
48+
--------------------------------------------------------
49+
`./bin/elasticsearch -Enode.attr.rack_id=rack_one`
50+
--------------------------------------------------------
3451

52+
. Tell {es} to take one or more awareness attributes into account when
53+
allocating shards by setting
54+
`cluster.routing.allocation.awareness.attributes` in *every* master-eligible
55+
node's `elasticsearch.yml` config file.
56+
+
57+
--
3558
[source,yaml]
3659
--------------------------------------------------------
37-
cluster.routing.allocation.awareness.attributes: rack_id
60+
cluster.routing.allocation.awareness.attributes: rack_id <1>
3861
--------------------------------------------------------
39-
40-
With this config in place, let's say we start two nodes with
41-
`node.attr.rack_id` set to `rack_one`, and we create an index with 5 primary
42-
shards and 1 replica of each primary. All primaries and replicas are
62+
<1> Specify multiple attributes as a comma-separated list.
63+
--
64+
+
65+
You can also use the
66+
<<cluster-update-settings,cluster-update-settings>> API to set or update
67+
a cluster's awareness attributes.
68+
69+
With this example configuration, if you start two nodes with
70+
`node.attr.rack_id` set to `rack_one` and create an index with 5 primary
71+
shards and 1 replica of each primary, all primaries and replicas are
4372
allocated across the two nodes.
4473

45-
Now, if we start two more nodes with `node.attr.rack_id` set to `rack_two`,
46-
Elasticsearch will move shards across to the new nodes, ensuring (if possible)
47-
that no two copies of the same shard will be in the same rack. However if
48-
`rack_two` were to fail, taking down both of its nodes, Elasticsearch will
49-
still allocate the lost shard copies to nodes in `rack_one`.
50-
51-
.Prefer local shards
52-
*********************************************
53-
54-
When executing search or GET requests, with shard awareness enabled,
55-
Elasticsearch will prefer using local shards -- shards in the same awareness
56-
group -- to execute the request. This is usually faster than crossing between
57-
racks or across zone boundaries.
74+
If you add two nodes with `node.attr.rack_id` set to `rack_two`,
75+
{es} moves shards to the new nodes, ensuring (if possible)
76+
that no two copies of the same shard are in the same rack.
5877

59-
*********************************************
60-
61-
Multiple awareness attributes can be specified, in which case each attribute
62-
is considered separately when deciding where to allocate the shards.
63-
64-
[source,yaml]
65-
-------------------------------------------------------------
66-
cluster.routing.allocation.awareness.attributes: rack_id,zone
67-
-------------------------------------------------------------
68-
69-
NOTE: When using awareness attributes, shards will not be allocated to nodes
70-
that don't have values set for those attributes.
71-
72-
NOTE: Number of primary/replica of a shard allocated on a specific group of
73-
nodes with the same awareness attribute value is determined by the number of
74-
attribute values. When the number of nodes in groups is unbalanced and there
75-
are many replicas, replica shards may be left unassigned.
78+
If `rack_two` fails and takes down both its nodes, by default {es}
79+
allocates the lost shard copies to nodes in `rack_one`. To prevent multiple
80+
copies of a particular shard from being allocated in the same location, you can
81+
enable forced awareness.
7682

7783
[float]
7884
[[forced-awareness]]
79-
=== Forced Awareness
80-
81-
Imagine that you have two zones and enough hardware across the two zones to
82-
host all of your primary and replica shards. But perhaps the hardware in a
83-
single zone, while sufficient to host half the shards, would be unable to host
84-
*ALL* the shards.
85+
==== Forced awareness
8586

86-
With ordinary awareness, if one zone lost contact with the other zone,
87-
Elasticsearch would assign all of the missing replica shards to a single zone.
88-
But in this example, this sudden extra load would cause the hardware in the
89-
remaining zone to be overloaded.
87+
By default, if one location fails, Elasticsearch assigns all of the missing
88+
replica shards to the remaining locations. While you might have sufficient
89+
resources across all locations to host your primary and replica shards, a single
90+
location might be unable to host *ALL* of the shards.
9091

91-
Forced awareness solves this problem by *NEVER* allowing copies of the same
92-
shard to be allocated to the same zone.
92+
To prevent a single location from being overloaded in the event of a failure,
93+
you can set `cluster.routing.allocation.awareness.force` so no replicas are
94+
allocated until nodes are available in another location.
9395

94-
For example, lets say we have an awareness attribute called `zone`, and we
95-
know we are going to have two zones, `zone1` and `zone2`. Here is how we can
96-
force awareness on a node:
96+
For example, if you have an awareness attribute called `zone` and configure nodes
97+
in `zone1` and `zone2`, you can use forced awareness to prevent Elasticsearch
98+
from allocating replicas if only one zone is available:
9799

98100
[source,yaml]
99101
-------------------------------------------------------------------
100-
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2 <1>
101102
cluster.routing.allocation.awareness.attributes: zone
103+
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2 <1>
102104
-------------------------------------------------------------------
103-
<1> We must list all possible values that the `zone` attribute can have.
104-
105-
Now, if we start 2 nodes with `node.attr.zone` set to `zone1` and create an
106-
index with 5 shards and 1 replica. The index will be created, but only the 5
107-
primary shards will be allocated (with no replicas). Only when we start more
108-
nodes with `node.attr.zone` set to `zone2` will the replicas be allocated.
109-
110-
The `cluster.routing.allocation.awareness.*` settings can all be updated
111-
dynamically on a live cluster with the
112-
<<cluster-update-settings,cluster-update-settings>> API.
113-
105+
<1> Specify all possible values for the awareness attribute.
114106

107+
With this example configuration, if you start two nodes with `node.attr.zone` set
108+
to `zone1` and create an index with 5 shards and 1 replica, Elasticsearch creates
109+
the index and allocates the 5 primary shards but no replicas. Replicas are
110+
only allocated once nodes with `node.attr.zone` set to `zone2` are available.

0 commit comments

Comments
 (0)