Skip to content

Commit 3f51fed

Browse files
pugnascotiaDaveCTurnerAdam Locke
authored
Dangling indices documentation (elastic#58751)
Part of elastic#48366. Add documentation for the dangling indices API added in elastic#58176. Co-authored-by: David Turner <[email protected]> Co-authored-by: Adam Locke <[email protected]>
1 parent 91db0db commit 3f51fed

7 files changed

+203
-7
lines changed

‎docs/reference/indices.asciidoc

+14
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,14 @@ index settings, aliases, mappings, and index templates.
7676
* <<indices-flush>>
7777
* <<indices-forcemerge>>
7878

79+
[float]
80+
[[dangling-indices-api]]
81+
=== Dangling indices:
82+
* <<dangling-indices-list>>
83+
* <<dangling-index-import>>
84+
* <<dangling-index-delete>>
85+
86+
7987

8088
include::indices/add-alias.asciidoc[]
8189

@@ -156,3 +164,9 @@ include::indices/aliases.asciidoc[]
156164
include::indices/update-settings.asciidoc[]
157165

158166
include::indices/resolve.asciidoc[]
167+
168+
include::indices/dangling-indices-list.asciidoc[]
169+
170+
include::indices/dangling-index-import.asciidoc[]
171+
172+
include::indices/dangling-index-delete.asciidoc[]

‎docs/reference/indices/dangling-index-delete.asciidoc

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
[[dangling-index-delete]]
2+
=== Delete dangling index API
3+
++++
4+
<titleabbrev>Delete dangling index</titleabbrev>
5+
++++
6+
7+
Deletes a dangling index.
8+
9+
[[dangling-index-delete-api-request]]
10+
==== {api-request-title}
11+
12+
[source,console]
13+
--------------------------------------------------
14+
DELETE /_dangling/<index-uuid>?accept_data_loss=true
15+
--------------------------------------------------
16+
// TEST[skip:Difficult to set up]
17+
18+
[[dangling-index-delete-api-desc]]
19+
==== {api-description-title}
20+
21+
include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
22+
23+
24+
Deletes a dangling index by referencing its UUID. Use the
25+
<<dangling-indices-list,List dangling indices API>> to locate the UUID of an index.
26+
27+
28+
[[dangling-index-delete-api-path-params]]
29+
==== {api-path-parms-title}
30+
31+
`<index-uuid>`::
32+
(Required, string)
33+
UUID of the index to delete. You can find this using the
34+
<<dangling-indices-list,List dangling indices API>>.
35+
36+
[[dangling-index-delete-api-query-params]]
37+
==== {api-query-parms-title}
38+
39+
`accept_data_loss`::
40+
(Optional, boolean)
41+
This field must be set to `true` in order to carry out the import, since it will
42+
no longer be possible to recover the data from the dangling index.
43+
44+
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]

‎docs/reference/indices/dangling-index-import.asciidoc

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
[[dangling-index-import]]
2+
=== Import dangling index API
3+
++++
4+
<titleabbrev>Import dangling index</titleabbrev>
5+
++++
6+
7+
Imports a dangling index.
8+
9+
[[dangling-index-import-api-request]]
10+
==== {api-request-title}
11+
12+
[source,console]
13+
--------------------------------------------------
14+
POST /_dangling/<index-uuid>?accept_data_loss=true
15+
--------------------------------------------------
16+
// TEST[skip:Difficult to set up]
17+
18+
[[dangling-index-import-api-desc]]
19+
==== {api-description-title}
20+
21+
include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
22+
23+
Import a single index into the cluster by referencing its UUID. Use the
24+
<<dangling-indices-list,List dangling indices API>> to locate the UUID of an index.
25+
26+
27+
[[dangling-index-import-api-path-params]]
28+
==== {api-path-parms-title}
29+
30+
`<index-uuid>`::
31+
(Required, string)
32+
UUID of the index to import, which you can find using the
33+
<<dangling-indices-list,List dangling indices API>>.
34+
35+
[[dangling-index-import-api-query-params]]
36+
==== {api-query-parms-title}
37+
38+
`accept_data_loss`::
39+
(Required, boolean)
40+
This field must be set to `true` to import a dangling index. Because {es}
41+
cannot know where the dangling index data came from or determine which shard
42+
copies are fresh and which are stale, it cannot guarantee that the imported data
43+
represents the latest state of the index when it was last in the cluster.
44+
45+
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]
46+
47+
[[dangling-index-import-api-example]]
48+
==== {api-examples-title}
49+
50+
The following example shows how to import a dangling index:
51+
52+
[source,console]
53+
--------------------------------------------------
54+
POST /_dangling/zmM4e0JtBkeUjiHD-MihPQ?accept_data_loss=true
55+
--------------------------------------------------
56+
// TEST[skip:Difficult to set up]
57+
58+
The API returns following response:
59+
60+
[source,console-result]
61+
--------------------------------------------------
62+
{
63+
"acknowledged" : true
64+
}
65+
--------------------------------------------------

‎docs/reference/indices/dangling-indices-list.asciidoc

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
[[dangling-indices-list]]
2+
=== List dangling indices API
3+
++++
4+
<titleabbrev>List dangling indices</titleabbrev>
5+
++++
6+
7+
Lists dangling indices.
8+
9+
[[dangling-indices-list-api-request]]
10+
==== {api-request-title}
11+
12+
[source,console]
13+
--------------------------------------------------
14+
GET /_dangling
15+
--------------------------------------------------
16+
// TEST[skip:TBD]
17+
18+
[[dangling-indices-list-api-desc]]
19+
==== {api-description-title}
20+
21+
// tag::dangling-index-description[]
22+
If {es} encounters index data that is absent from the current cluster
23+
state, those indices are considered to be dangling. For example,
24+
this can happen if you delete more than
25+
`cluster.indices.tombstones.size` indices while an {es} node is offline.
26+
// end::dangling-index-description[]
27+
28+
Use this API to list dangling indices, which you can then
29+
<<dangling-index-import,import>> or <<dangling-index-delete,delete>>.
30+
31+
32+
[[dangling-indices-list-api-example]]
33+
==== {api-examples-title}
34+
35+
The API returns the following response:
36+
37+
[source,console-result]
38+
--------------------------------------------------
39+
{
40+
"dangling_indices": [
41+
"index_name": "twitter",
42+
"index_uuid": "zmM4e0JtBkeUjiHD-MihPQ",
43+
"creation_date_millis": 1589414451372,
44+
"node_ids": [
45+
"pL47UN3dAb2d5RCWP6lQ3e"
46+
]
47+
]
48+
}
49+
--------------------------------------------------

‎docs/reference/modules/cluster/misc.asciidoc

+4
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,10 @@ increase it if you expect nodes to be absent from the cluster and miss more
107107
than 500 deletes. We think that is rare, thus the default. Tombstones don't take
108108
up much space, but we also think that a number like 50,000 is probably too big.
109109

110+
include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
111+
You can use the <<dangling-indices-api,Dangling indices API>> to manage
112+
this situation.
113+
110114
[[cluster-logger]]
111115
===== Logger
112116

‎docs/reference/modules/discovery.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ formation:
5959
<<modules-discovery-settings,Settings>>::
6060

6161
There are settings that enable users to influence the discovery, cluster
62-
formation, master election and fault detection processes.
62+
formation, master election and fault detection processes.
6363

6464
include::discovery/discovery.asciidoc[]
6565

‎docs/reference/modules/gateway.asciidoc

+26-6
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,29 @@ NOTE: These settings only take effect on a full cluster restart.
5656
[[dangling-indices]]
5757
==== Dangling indices
5858

59-
When a node joins the cluster, any shards stored in its local data
60-
directory which do not already exist in the cluster will be imported into the
61-
cluster. This functionality is intended as a best effort to help users who
62-
lose all master nodes. If a new master node is started which is unaware of
63-
the other indices in the cluster, adding the old nodes will cause the old
64-
indices to be imported, instead of being deleted.
59+
When a node joins the cluster, if it finds any shards stored in its local data
60+
directory that do not already exist in the cluster, it will consider those
61+
shards to be "dangling". Importing dangling indices
62+
into the cluster using `gateway.auto_import_dangling_indices` is not safe.
63+
Instead, use the <<dangling-indices-api,Dangling indices API>>. Neither
64+
mechanism provides any guarantees as to whether the imported data truly
65+
represents the latest state of the data when the index was still part of
66+
the cluster.
67+
68+
`gateway.auto_import_dangling_indices`::
69+
70+
deprecated:[7.9.0, This setting will be removed in 8.0. You should use the dedicated dangling indices API instead.]
71+
Whether to automatically import dangling indices into the cluster
72+
state, provided no indices already exist with the same name. Defaults
73+
to `false`.
74+
75+
WARNING: The auto-import functionality was intended as a best effort to help users
76+
who lose all master nodes. For example, if a new master node were to be
77+
started which was unaware of the other indices in the cluster, adding the
78+
old nodes would cause the old indices to be imported, instead of being
79+
deleted. However there are several issues with automatic importing, and
80+
its use is strongly discouraged in favour of the
81+
<<dangling-indices-api,dedicated API>.
82+
83+
WARNING: Losing all master nodes is a situation that should be avoided at
84+
all costs, as it puts your cluster's metadata and data at risk.

0 commit comments

Comments
 (0)