Skip to content

Commit c8e3128

Browse files
debadairgchapslcawlMelori Arellanojrodewig
authored
[DOCS] Combo version of ILM docs. (#57909) (#59029)
* [DOCS] Combo version of ILM docs. * [DOCS] Moved tutorial from Kibana. * Adds documentation for index lifecycle policies (#28705) * [DOCS] Adds documentation for index lifecycle policies * [DOCS] Updated image for policy options to show all menu items * Update create-policy.asciidoc * [DOCS] Incorporated review comments on hot and warm phase * [DOCS] Additional changes to warm phase * [DOCS] Removed the word open in the warm phase * Adds X-Pack icon for ILM (#34178) * Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <[email protected]> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <[email protected]> * Updates for navigation redesign (#68709) * [DOCS] Updates for navigation redesign * Getting started * Set up text * Discover * Dashboard, Graph, ML, Maps, APM, SIEM, Dev tools * Dev Tools, Stack Monitoring, Management * Management * Final changes * [DOCS] Updates for navigation redesign * [DOCS] Updates CCR monitoring screenshots * updates SIEM screenshot and Cases overview text * Added Brandon's APM image * [DOCS] Refines CCR shard screenshot * Removed merge conflict image file Co-authored-by: lcawl <[email protected]> Co-authored-by: Ben Skelker <[email protected]> * [DOCS] Put API examples in collapsible sections like ML does * Fix include * Added tutorial images * Fixed images * Add short title for FB tutorial * Add missing files * Incorporate review feedback * review feedback * Incorporated review feedback Co-authored-by: gchaps <[email protected]> Co-authored-by: Lisa Cawley <[email protected]> Co-authored-by: Melori Arellano <[email protected]> Co-authored-by: James Rodewig <[email protected]> Co-authored-by: Kaarina Tungseth <[email protected]> Co-authored-by: Ben Skelker <[email protected]> Co-authored-by: gchaps <[email protected]> Co-authored-by: Lisa Cawley <[email protected]> Co-authored-by: Melori Arellano <[email protected]> Co-authored-by: James Rodewig <[email protected]> Co-authored-by: Kaarina Tungseth <[email protected]> Co-authored-by: Ben Skelker <[email protected]>
1 parent 7c43cbc commit c8e3128

16 files changed

+257
-54
lines changed

‎docs/reference/glossary.asciidoc

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -268,7 +268,6 @@ shard will never be started on the same node as its primary shard.
268268
--
269269
// tag::rollover-def[]
270270
// tag::rollover-def-short[]
271-
272271
Creates a new index for a rollover target when the existing index reaches a certain size, number of docs, or age.
273272
A rollover target can be either an <<indices-aliases, index alias>> or a <<data-streams, data stream>>.
274273
// end::rollover-def-short[]

‎docs/reference/ilm/example-index-lifecycle-policy.asciidoc

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
[role="xpack"]
2+
3+
[[example-using-index-lifecycle-policy]]
4+
=== Tutorial: Manage {filebeat} time-based indices
5+
++++
6+
<titleabbrev>Manage {filebeat} time-based indices</titleabbrev>
7+
++++
8+
9+
With {ilm} ({ilm-init}), you can create policies that perform actions automatically
10+
on indices as they age and grow. {ilm-init} policies help you to manage
11+
performance, resilience, and retention of your data during its lifecycle. This tutorial shows
12+
you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init}
13+
policies. You can learn more about all of the actions, benefits, and lifecycle
14+
phases in the <<overview-index-lifecycle-management, {ilm-init} overview>>.
15+
16+
17+
[discrete]
18+
[[example-using-index-lifecycle-policy-scenario]]
19+
==== Scenario
20+
21+
You’re tasked with sending syslog files to an {es} cluster. This
22+
log data has the following data retention guidelines:
23+
24+
* Keep logs on hot data nodes for 30 days
25+
* Roll over to a new index if the size reaches 50GB
26+
* After 30 days:
27+
** Move the logs to warm data nodes
28+
** Set <<glossary-replica-shard, replica shards>> to 1
29+
** <<indices-forcemerge, Force merge>> multiple index segments to free up the space used by deleted documents
30+
* Delete logs after 90 days
31+
32+
33+
[discrete]
34+
[[example-using-index-lifecycle-policy-prerequisites]]
35+
==== Prerequisites
36+
37+
To complete this tutorial, you'll need:
38+
39+
* An {es} cluster with hot and warm nodes configured for shard allocation
40+
awareness.
41+
42+
** {ess}:
43+
Choose the {cloud}/ec-getting-started-templates-hot-warm.html[hot-warm architecture] deployment template.
44+
45+
** Self-managed cluster:
46+
Add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering].
47+
+
48+
For example, you can set this in your `elasticsearch.yml` for each data node:
49+
+
50+
[source,yaml]
51+
--------------------------------------------------------------------------------
52+
node.attr.data: "warm"
53+
--------------------------------------------------------------------------------
54+
55+
* A server with {filebeat} installed and configured to send logs to the `elasticsearch`
56+
output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}].
57+
58+
[discrete]
59+
[[example-using-index-lifecycle-policy-view-fb-ilm-policy]]
60+
==== View the {filebeat} {ilm-init} policy
61+
62+
{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init}
63+
is enabled automatically if you’re using the default `filebeat.yml` and index template.
64+
65+
To view the default policy in {kib}:
66+
67+
. Go to Management and select *Index Lifecycle Policies*.
68+
. Search for _filebeat_
69+
. Select the _filebeat-version_ policy.
70+
71+
This policy initiates the rollover action when the index size reaches 50GB or
72+
becomes 30 days old.
73+
74+
[role="screenshot"]
75+
image::images/ilm/tutorial-ilm-hotphaserollover-default.png["Default policy"]
76+
77+
78+
[discrete]
79+
==== Modify the policy
80+
81+
The default policy is enough to prevent the creation of many tiny daily indices.
82+
You can modify the policy to meet more complex requirements.
83+
84+
. Activate the warm phase.
85+
+
86+
--
87+
[role="screenshot"]
88+
image::images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"]
89+
90+
.. Set one of the following options to control when the index moves to the warm phase:
91+
92+
*** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the
93+
indices on hot nodes for a range of 15-45 days, depending on when the initial
94+
rollover occurred.
95+
96+
*** Enable *Move to warm phase on rollover*. The index might move to the warm phase
97+
more quickly than intended if it reaches the *Maximum index size* before the
98+
the *Maximum age*.
99+
100+
.. In the *Select a node attribute to control shard allocation* dropdown, select
101+
*data:warm(2)* to migrate shards to warm data nodes.
102+
103+
.. Change *Number of replicas* to *1*.
104+
105+
.. Enable *Force merge data* and set *Number of segments* to *1*.
106+
107+
NOTE: When rollover is enabled in the hot phase, action timing in the other phases
108+
is based on the rollover date.
109+
--
110+
111+
. Activate the delete phase and set *Timing for delete phase* to *90* days.
112+
+
113+
[role="screenshot"]
114+
image::images/ilm/tutorial-ilm-delete-rollover.png["Add a delete phase"]
115+
116+
[discrete]
117+
==== Create a custom policy
118+
119+
If meeting a specific retention time period is most important, you can create a
120+
custom policy. For this option, you use {filebeat} daily indices without
121+
rollover.
122+
123+
To create a custom policy:
124+
125+
. Go to Management and select *Index Lifecycle Policies*.
126+
. Click *Create policy*.
127+
. Activate the warm phase and configure it as follows:
128+
+
129+
--
130+
**Timing for warm phase**: 30 days from index creation
131+
132+
**Node attribute**: `data:warm`
133+
134+
**Number of replicas**: 1
135+
136+
**Force merge data**: enable
137+
138+
**Number of segments**: 1
139+
140+
[role="screenshot"]
141+
image::images/ilm/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"]
142+
--
143+
144+
. Activate the delete phase and set the timing to 90 days.
145+
+
146+
[role="screenshot"]
147+
image::images/ilm/tutorial-ilm-delete-phase-creation.png["Delete phase"]
148+
149+
To configure the index to use the new policy:
150+
151+
. Go to Management and select *Index Lifecycle Policies*.
152+
. Find your {ilm-init} policy and click its *Actions* link.
153+
. Choose *Add policy to index template*.
154+
. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`.
155+
. Click *Add Policy* to save the changes.
156+
+
157+
NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will
158+
see a notice that the template already has a policy associated with it. Confirm
159+
that you want to overwrite that configuration.
160+
161+
When you change the policy associated with the index template, the active
162+
index will continue to use the policy it was associated with at index creation
163+
unless you manually update it. The next new index will use the updated policy.
164+
For more reasons that your {ilm-init} policy changes might be delayed, see
165+
<<update-lifecycle-policy, Update Lifecycle Policy>>.

‎docs/reference/ilm/ilm-overview.asciidoc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices
1111
according to your performance, resiliency, and retention requirements.
12+
1213
Index lifecycle policies can trigger actions such as:
1314

1415
* **Rollover**:
@@ -39,9 +40,9 @@ For example, if you are indexing metrics data from a fleet of ATMs into
3940
Elasticsearch, you might define a policy that says:
4041

4142
. When the index reaches 50GB, roll over to a new index.
42-
. Move the old index into the warm stage, mark it read only, and shrink it down
43+
. Move the old index into the warm phase, mark it read only, and shrink it down
4344
to a single shard.
44-
. After 7 days, move the index into the cold stage and move it to less expensive
45+
. After 7 days, move the index into the cold phase and move it to less expensive
4546
hardware.
4647
. Delete the index once the required 30 day retention period is reached.
4748

‎docs/reference/ilm/ilm-tutorial.asciidoc

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,6 @@
77
<titleabbrev>Automate rollover</titleabbrev>
88
++++
99

10-
This tutorial demonstrates how to use {ilm}
11-
({ilm-init}) to manage indices that contain time-series data.
12-
1310
When you continuously index timestamped documents into {es},
1411
you typically use a <<data-streams, data stream>> so you can periodically roll over to a
1512
new index.
@@ -51,18 +48,23 @@ A lifecycle policy specifies the phases in the index lifecycle
5148
and the actions to perform in each phase. A lifecycle can have up to four phases:
5249
`hot`, `warm`, `cold`, and `delete`.
5350

54-
You can define and manage policies through {kib} Management or with the
55-
<<ilm-put-lifecycle, put policy>> API.
56-
5751
For example, you might define a `timeseries_policy` that has two phases:
5852

5953
* A `hot` phase that defines a rollover action to specify that an index rolls over when it
6054
reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
6155
* A `delete` phase that sets `min_age` to remove the index 90 days after rollover.
6256
Note that this value is relative to the rollover time, not the index creation time.
6357

64-
The underlying put policy request looks like this:
58+
You can create the policy through {kib} Management or with the
59+
<<ilm-put-lifecycle, put policy>> API.
60+
To create the policy from {kib}, go to Management and click **Index Lifecycle Policies**.
61+
62+
[role="screenshot"]
63+
image:images/ilm/create-policy.png[]
6564

65+
.API example
66+
[%collapsible]
67+
====
6668
[source,console]
6769
------------------------
6870
PUT _ilm/policy/timeseries_policy
@@ -91,10 +93,7 @@ PUT _ilm/policy/timeseries_policy
9193
<2> Trigger the `rollover` action when either of the conditions are met.
9294
<3> Move the index into the `delete` phase 90 days after rollover.
9395
<4> Trigger the `delete` action when the index enters the delete phase.
94-
95-
You can also invoke this API directly to add lifecycle policies.
96-
97-
For the complete list of actions that {ilm} can perform, see <<ilm-actions>>.
96+
====
9897

9998
[discrete]
10099
[[ilm-gs-apply-policy]]
@@ -114,8 +113,9 @@ You can use the {kib} Create template wizard to add the template.
114113
This wizard invokes the put _index_template API to create the <<indices-templates,index template>>
115114
with the options you specify.
116115

117-
The underlying request looks like this:
118-
116+
.API example
117+
[%collapsible]
118+
====
119119
[source,console]
120120
-----------------------
121121
PUT _index_template/timeseries_template
@@ -148,8 +148,7 @@ in all documents indexed into the `timeseries` data stream.
148148
<3> The name of the {ilm-init} policy used to manage the data stream.
149149
<4> A <<date,`date`>> or <<date_nanos,`date_nanos`>> field mapping for the
150150
timestamp field specified in the `timestamp_field` property
151-
152-
You can also invoke this API directly to add templates.
151+
====
153152

154153
[discrete]
155154
[[ilm-gs-create-the-data-stream]]
@@ -317,10 +316,14 @@ that match the index pattern.
317316
* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
318317
when the rollover action is triggered for an index.
319318

320-
You can use the {kib} Create template wizard to add the template.
321-
This wizard invokes the put template API to create the template with the options you specify.
319+
You can use the {kib} Create template wizard to add the template.
320+
To access the wizard, go to Management, click **Index Management**,
321+
and select the **Index Templates** view.
322+
323+
[role="screenshot"]
324+
image:images/ilm/create-template-wizard.png[]
322325

323-
The underlying request looks like this:
326+
The create template request for the example template looks like this:
324327

325328
[source,console]
326329
-----------------------
@@ -342,9 +345,6 @@ PUT _template/timeseries_template
342345
<3> The name of the alias used to reference these indices.
343346
Required for policies that use the rollover action.
344347

345-
You can also invoke this API directly to add templates.
346-
347-
348348
//////////////////////////
349349
350350
[source,console]

‎docs/reference/ilm/ilm-with-existing-indices.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ but a combined index needs to be retained until you are ready to delete the new
6868

6969
. Reduce the {ilm-init} poll interval to ensure that the index doesn't
7070
grow too large while waiting for the rollover check.
71-
By default, {ilm-init} checks rollover conditions every 10 minutes.
71+
By default, {ilm-init} checks to see what actions need to be taken every 10 minutes.
7272
+
7373
--
7474
[source,console]

‎docs/reference/ilm/index.asciidoc

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,20 @@
55

66
[partintro]
77
--
8-
You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to
9-
your performance, resiliency, and retention requirements.
8+
You can configure {ilm} ({ilm-init}) policies to automatically manage indices
9+
according to your performance, resiliency, and retention requirements.
1010
For example, you could use {ilm-init} to:
1111

1212
* Spin up a new index when an index reaches a certain size or number of documents
1313
* Create a new index each day, week, or month and archive previous ones
1414
* Delete stale indices to enforce data retention standards
15-
15+
16+
You can create and manage index lifecycle policies through {kib} Management or the {ilm-init} APIs.
1617
When you enable {ilm} for {beats} or the {ls} {es} output plugin,
17-
{ilm-init} is configured automatically.
18-
You can modify the default policies through {kib} Management or the {ilm-init} APIs.
18+
default policies are configured automatically.
19+
20+
[role="screenshot"]
21+
image:images/ilm/index-lifecycle-policies.png[]
1922

2023
[TIP]
2124
To automatically back up your indices and manage snapshots,
@@ -24,13 +27,15 @@ use <<getting-started-snapshot-lifecycle-management,snapshot lifecycle policies>
2427
* <<overview-index-lifecycle-management>>
2528
* <<ilm-concepts>>
2629
* <<getting-started-index-lifecycle-management>>
27-
* <<ilm-actions>>
30+
* <<example-using-index-lifecycle-policy>>
2831
* <<set-up-lifecycle-policy>>
2932
* <<index-lifecycle-error-handling>>
3033
* <<start-stop-ilm>>
3134
* <<ilm-with-existing-indices>>
3235
* <<skipping-rollover>>
3336
* <<index-lifecycle-and-snapshots>>
37+
* <<index-lifecycle-management-api>>
38+
* <<ilm-actions>>
3439

3540
--
3641
include::ilm-overview.asciidoc[]
@@ -39,6 +44,8 @@ include::ilm-concepts.asciidoc[]
3944

4045
include::ilm-tutorial.asciidoc[]
4146

47+
include::example-index-lifecycle-policy.asciidoc[leveloffset=-1]
48+
4249
include::ilm-actions.asciidoc[]
4350

4451
include::set-up-lifecycle-policy.asciidoc[]

0 commit comments

Comments
 (0)