Skip to content

Commit f70d447

Browse files
talevytalevy
authored
[ILM] getting started documentation (#35379)
1 parent 8ea999e commit f70d447

File tree

1 file changed

+220
-7
lines changed

1 file changed

+220
-7
lines changed

‎docs/reference/ilm/getting-started-ilm.asciidoc

Lines changed: 220 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,228 @@
11
[role="xpack"]
2+
[testenv="basic"]
23
[[getting-started-index-lifecycle-management]]
34
== Getting started with {ilm}
45

5-
Create a policy that rolls over after 1 day deletes an index after 30 days
6+
Let's jump into {ILM} by working through a hands-on scenario.
7+
This section will leverage many new concepts unique to {ILM} that
8+
you may not be familiar with. The following sections will explore
9+
these in more details.
610

7-
Show create policy API req/res
11+
The goal of this example is to set up a set of indices that will encapsulate
12+
the data from a time series data source. We can imagine there is a system
13+
like {filebeat-ref}[Filebeat] that continuously indexes documents into
14+
our writing index. We wish to roll over the index after it reaches a size
15+
of 50 gigabytes, or has been created 30 days ago, and then delete the index
16+
after 90 days.
817

9-
Show assign policy to index API req/res
18+
=== Setting up a new policy
1019

11-
Show both the API and how it is done with `index.lifecyce.name` using the
12-
create-index API
20+
There are many new features introduced by {ILM}, but we will only focus on
21+
a few that are needed for our example. For starters, we will use the
22+
<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
23+
policies are defined in JSON and include specific
24+
<<ilm-policy-definition,phases and actions>>.
1325

14-
Show explain API to show current state, but ignore the “step” related info,
15-
only focus on managed/phase/action
26+
[source,js]
27+
------------------------
28+
PUT _ilm/policy/datastream_policy <1>
29+
{
30+
"policy": { <2>
31+
"phases": {
32+
"hot": { <3>
33+
"actions": {
34+
"rollover": { <4>
35+
"max_size": "50GB",
36+
"max_age": "30d"
37+
}
38+
}
39+
},
40+
"delete": {
41+
"min_age": "90d", <5>
42+
"actions": {
43+
"delete": {} <6>
44+
}
45+
}
46+
}
47+
}
48+
}
49+
------------------------
50+
// CONSOLE
51+
// TEST
52+
<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
53+
a new policy named "datastream_policy"
54+
<2> policy definition sub-object
55+
<3> the hot phase defined in the "phases" section. Optional `min_age` field
56+
not defined -- defaults to `0ms`
57+
<4> rollover action definition
58+
<5> delete phase begins after 90 days
59+
<6> delete action definition
60+
61+
62+
Here we created the policy called `datastream_policy` which rolls over
63+
the index being written to after it reaches 50 gigabytes, or it is 30
64+
days old. The rollover will occur when either of these conditions is true.
65+
The index will be deleted 90 days after it is rolled over.
66+
67+
=== Applying a policy to our index
68+
69+
There are <<set-up-lifecycle-policy,a few ways>> to associate a
70+
policy to an index. Since we wish specific settings to be applied to
71+
the new index created from Rollover, we will set the policy via
72+
index templates.
73+
74+
75+
[source,js]
76+
-----------------------
77+
PUT _template/datastream_template
78+
{
79+
"index_patterns": ["datastream-*"], <1>
80+
"settings": {
81+
"number_of_shards": 1,
82+
"number_of_replicas": 1,
83+
"index.lifecycle.name": "datastream_policy", <2>
84+
"index.lifecycle.rollover_alias": "datastream" <3>
85+
}
86+
}
87+
-----------------------
88+
// CONSOLE
89+
// TEST[continued]
90+
<1> match all indices starting with "datastream-". These will include all
91+
newly created indices from actions like rollover
92+
<2> the name of the lifecycle policy managing the index
93+
<3> alias to use for the rollover action, required since a rollover action is
94+
defined in the policy.
95+
96+
The above index template introduces a few new settings specific to {ILM}.
97+
The first being `index.lifecycle.name`. This setting will configure
98+
the "datastream_policy" to the index applying this template. This means
99+
that all newly created indices prefixed "datastream-" will be managed by
100+
our policy. The other setting used here is `index.lifecycle.rollover_alias`.
101+
This setting is required when using a policy containing the rollover
102+
action and specifies which alias to rollover on behalf of this index.
103+
The intention here is that the rollover alias is also defined on the index.
104+
105+
To begin, we will want to bootstrap our first index to write to.
106+
107+
108+
[source,js]
109+
-----------------------
110+
PUT datastream-000001
111+
{
112+
"aliases": {
113+
"datastream": {
114+
"is_write_index": true
115+
}
116+
}
117+
}
118+
-----------------------
119+
// CONSOLE
120+
// TEST[continued]
121+
122+
When creating our index, we have to consider a few important configurations
123+
that tie our index and our policy together correctly. We need to make sure
124+
that our index name matches our index template pattern of "datastream-*",
125+
which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
126+
requires that our index name ends with a number. In our case, we used
127+
`000001`. This is important so that Rollover can increment this number when
128+
naming the new index created from rolling over.
129+
130+
Our index creation request leverages its template to apply our settings,
131+
but we must also configure our rollover alias: "datastream". To do this,
132+
we take advantage of <<aliases-write-index,write indices>>. This is a way
133+
to define an alias to be used for both reading and writing, with only one
134+
index being the index that is being written to at a time. Rollover swaps
135+
the write index to be the new index created from rollover, and sets the
136+
alias to be read-only for the source index.
137+
138+
=== Checking progress
139+
140+
Now that we have an index managed by our policy, how do we tell what is going
141+
on? Which phase are we in? Is something broken? This section will go over a
142+
few APIs and their responses to help us inspect our indices with respect
143+
to {ILM}.
144+
145+
With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
146+
things like which phase we're in and when we entered that phase. The API
147+
will also provide further info if errors occurred, or if we are blocked on
148+
certain checks within actions.
149+
150+
[source,js]
151+
--------------------------------------------------
152+
GET datastream-*/_ilm/explain
153+
--------------------------------------------------
154+
// CONSOLE
155+
// TEST[continued]
156+
157+
The above request will retrieve {ILM} execution information for all our
158+
managed indices.
159+
160+
161+
[source,js]
162+
--------------------------------------------------
163+
{
164+
"indices": {
165+
"datastream-000001": {
166+
"index": "datastream-000001",
167+
"managed": true, <1>
168+
"policy": "datastream_policy", <2>
169+
"lifecycle_date_millis": 1538475653281,
170+
"phase": "hot", <3>
171+
"phase_time_millis": 1538475653317,
172+
"action": "rollover", <4>
173+
"action_time_millis": 1538475653317,
174+
"step": "attempt_rollover", <5>
175+
"step_time_millis": 1538475653317,
176+
"phase_execution": {
177+
"policy": "datastream_policy",
178+
"phase_definition": { <6>
179+
"min_age": "0ms",
180+
"actions": {
181+
"rollover": {
182+
"max_size": "50gb",
183+
"max_age": "30d"
184+
}
185+
}
186+
},
187+
"version": 1, <7>
188+
"modified_date_in_millis": 1539609701576
189+
}
190+
}
191+
}
192+
}
193+
--------------------------------------------------
194+
// CONSOLE
195+
// TESTRESPONSE[s/"lifecycle_date_millis": 1538475653281/"lifecycle_date_millis": $body.indices.datastream-000001.lifecycle_date_millis/]
196+
// TESTRESPONSE[s/"phase_time_millis": 1538475653317/"phase_time_millis": $body.indices.datastream-000001.phase_time_millis/]
197+
// TESTRESPONSE[s/"action_time_millis": 1538475653317/"action_time_millis": $body.indices.datastream-000001.action_time_millis/]
198+
// TESTRESPONSE[s/"step_time_millis": 1538475653317/"step_time_millis": $body.indices.datastream-000001.step_time_millis/]
199+
// TESTRESPONSE[s/"modified_date_in_millis": 1539609701576/"modified_date_in_millis": $body.indices.datastream-000001.phase_execution.modified_date_in_millis/]
200+
<1> this index is managed by ILM
201+
<2> the policy in question, in this case, "datastream_policy"
202+
<3> what phase the index is currently in
203+
<4> what action the index is currently on
204+
<5> what step the index is currently on
205+
<6> the definition of the phase
206+
(in this case, the "hot" phase) that the index is currently on
207+
<7> the version of the policy being used to execute the current phase
208+
209+
You can read about the full details of this response in the
210+
<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
211+
the response details which phase, action, and step we're in. We are in the
212+
"hot" phase, and "rollover" action. Rollover will continue to be called
213+
by {ILM} until its conditions are met and it rolls over the index.
214+
Afterwards, the original index will stay in the hot phase until 90 more
215+
days pass and it is deleted in the delete phase.
216+
As time goes on, new indices will be created and deleted.
217+
With `datastream-000002` being created when the index mets the rollover
218+
conditions and `datastream-000003` created after that. We will be able
219+
to search across all of our managed indices using the "datastream" alias,
220+
and we will be able to write to our to-be-rolled-over write indices using
221+
that same alias.
222+
223+
224+
225+
That's it! We have our first use-case managed by {ILM}.
226+
227+
To learn more about all our APIs,
228+
check out <<index-lifecycle-management-api,ILM APIs>>.

0 commit comments

Comments
 (0)