Backport ilm combo

docs/reference/ilm/example-index-lifecycle-policy.asciidoc

@@ -0,0 +1,165 @@
+[role="xpack"]
+
+[[example-using-index-lifecycle-policy]]
+=== Tutorial: Manage {filebeat} time-based indices
+++++
+<titleabbrev>Manage {filebeat} time-based indices</titleabbrev>
+++++
+
+With {ilm} ({ilm-init}), you can create policies that perform actions automatically 
+on indices as they age and grow. {ilm-init} policies help you to manage
+performance, resilience, and retention of your data during its lifecycle. This tutorial shows
+you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init}
+policies. You can learn more about all of the actions, benefits, and lifecycle
+phases in the <<overview-index-lifecycle-management, {ilm-init} overview>>.
+
+
+[discrete]
+[[example-using-index-lifecycle-policy-scenario]]
+==== Scenario
+
+You’re tasked with sending syslog files to an {es} cluster. This
+log data has the following data retention guidelines:
+
+* Keep logs on hot data nodes for 30 days
+* Roll over to a new index if the size reaches 50GB
+* After 30 days:
+** Move the logs to warm data nodes
+** Set <<glossary-replica-shard, replica shards>> to 1
+** <<indices-forcemerge, Force merge>> multiple index segments to free up the space used by deleted documents
+* Delete logs after 90 days
+
+
+[discrete]
+[[example-using-index-lifecycle-policy-prerequisites]]
+==== Prerequisites
+
+To complete this tutorial, you'll need:
+
+* An {es} cluster with hot and warm nodes configured for shard allocation
+awareness. 
+
+** {ess}: 
+Choose the {cloud}/ec-getting-started-templates-hot-warm.html[hot-warm architecture] deployment template.
+
+** Self-managed cluster: 
+Add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering].
++
+For example, you can set this in your `elasticsearch.yml` for each data node:
++
+[source,yaml]
+--------------------------------------------------------------------------------
+node.attr.data: "warm"
+--------------------------------------------------------------------------------
+
+* A server with {filebeat} installed and configured to send logs to the `elasticsearch`
+output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}].
+
+[discrete]
+[[example-using-index-lifecycle-policy-view-fb-ilm-policy]]
+==== View the {filebeat} {ilm-init} policy
+
+{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init}
+is enabled automatically if you’re using the default `filebeat.yml` and index template.
+
+To view the default policy in {kib}:
+
+. Go to Management and select *Index Lifecycle Policies*.
+. Search for _filebeat_
+. Select the _filebeat-version_ policy.
+
+This policy initiates the rollover action when the index size reaches 50GB or
+becomes 30 days old.
+
+[role="screenshot"]
+image::images/ilm/tutorial-ilm-hotphaserollover-default.png["Default policy"]
+
+
+[discrete]
+==== Modify the policy
+
+The default policy is enough to prevent the creation of many tiny daily indices.
+You can modify the policy to meet more complex requirements.
+
+. Activate the warm phase.
++
+--
+[role="screenshot"]
+image::images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"]
+
+.. Set one of the following options to control when the index moves to the warm phase:
+
+*** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the
+indices on hot nodes for a range of  15-45 days, depending on when the initial
+rollover occurred.
+
+*** Enable *Move to warm phase on rollover*. The index might move to the warm phase
+more quickly than intended if it reaches the *Maximum index size* before the
+the *Maximum age*.
+
+.. In the *Select a node attribute to control shard allocation* dropdown, select
+*data:warm(2)* to migrate shards to warm data nodes.
+
+.. Change *Number of replicas* to *1*.
+
+.. Enable *Force merge data* and set *Number of segments* to *1*.
+
+NOTE:  When rollover is enabled in the hot phase, action timing in the other phases
+is based on the rollover date.
+--
+
+. Activate the delete phase and set *Timing for delete phase* to *90* days.
++
+[role="screenshot"]
+image::images/ilm/tutorial-ilm-delete-rollover.png["Add a delete phase"]
+
+[discrete]
+==== Create a custom policy
+
+If meeting a specific retention time period is most important, you can create a
+custom policy.  For this option, you use {filebeat} daily indices without
+rollover.
+
+To create a custom policy:
+
+. Go to Management and select *Index Lifecycle Policies*.
+. Click *Create policy*.
+. Activate the warm phase and configure it as follows:
++
+--
+**Timing for warm phase**: 30 days from index creation
+
+**Node attribute**: `data:warm`
+
+**Number of replicas**: 1
+
+**Force merge data**: enable
+
+**Number of segments**: 1
+
+[role="screenshot"]
+image::images/ilm/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"]
+--
+
+. Activate the delete phase and set the timing to 90 days.
++
+[role="screenshot"]
+image::images/ilm/tutorial-ilm-delete-phase-creation.png["Delete phase"]
+
+To configure the index to use the new policy:
+
+. Go to Management and select *Index Lifecycle Policies*.
+. Find your {ilm-init} policy and click its *Actions* link.
+. Choose *Add policy to index template*.
+. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`.
+. Click *Add Policy* to save the changes.
++
+NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will
+see a notice that the template already has a policy associated with it. Confirm
+that you want to overwrite that configuration.
+
+When you change the policy associated with the index template, the active
+index will continue to use the policy it was associated with at index creation
+unless you manually update it. The next new index will use the updated policy.
+For more reasons that your {ilm-init} policy changes might be delayed, see
+<<update-lifecycle-policy, Update Lifecycle Policy>>.

docs/reference/ilm/ilm-overview.asciidoc

@@ -9,6 +9,7 @@
 
 You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices 
 according to your performance, resiliency, and retention requirements. 
+
 Index lifecycle policies can trigger actions such as:
 
 * **Rollover**: 
@@ -39,9 +40,9 @@ For example, if you are indexing metrics data from a fleet of ATMs into
 Elasticsearch, you might define a policy that says:
 
 . When the index reaches 50GB, roll over to a new index.
-. Move the old index into the warm stage, mark it read only, and shrink it down
+. Move the old index into the warm phase, mark it read only, and shrink it down
 to a single shard.
-. After 7 days, move the index into the cold stage and move it to less expensive
+. After 7 days, move the index into the cold phase and move it to less expensive
 hardware.
 . Delete the index once the required 30 day retention period is reached.
 

docs/reference/ilm/ilm-tutorial.asciidoc

@@ -7,9 +7,6 @@
 <titleabbrev>Automate rollover</titleabbrev>
 ++++
 
-This tutorial demonstrates how to use {ilm} 
-({ilm-init}) to manage indices that contain time-series data.
-
 When you continuously index timestamped documents into {es},
 you typically use an index alias so you can periodically roll over to a new index.
 This enables you to implement a hot-warm-cold architecture to meet your performance
@@ -42,18 +39,23 @@ A lifecycle policy specifies the phases in the index lifecycle
 and the actions to perform in each phase. A lifecycle can have up to four phases:
 `hot`, `warm`, `cold`, and `delete`. 
 
-You can define and manage policies through {kib} Management or with the 
-<<ilm-put-lifecycle, put policy>> API.
-
 For example, you might define a `timeseries_policy` that has two phases:
 
 * A `hot` phase that defines a rollover action to specify that an index rolls over when it 
 reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
 * A `delete` phase that sets `min_age` to remove the index 90 days after rollover.
 Note that this value is relative to the rollover time, not the index creation time. 
 
-The underlying put policy request looks like this:
+You can create the policy through {kib} Management or with the 
+<<ilm-put-lifecycle, put policy>> API. 
+To create the policy from {kib}, go to Management and click **Index Lifecycle Policies**.
 
+[role="screenshot"]
+image:images/ilm/create-policy.png[]
+
+.API example
+[%collapsible]
+====
 [source,console]
 ------------------------
 PUT _ilm/policy/timeseries_policy
@@ -82,10 +84,7 @@ PUT _ilm/policy/timeseries_policy
 <2> Trigger the `rollover` action when either of the conditions are met.
 <3> Move the index into the `delete` phase 90 days after rollover.
 <4> Trigger the `delete` action when the index enters the delete phase.
-
-You can also invoke this API directly to add lifecycle policies.
-
-For the complete list of actions that {ilm} can perform, see <<ilm-actions>>.
+====
 
 [discrete]
 [[ilm-gs-apply-policy]]
@@ -107,7 +106,7 @@ when the rollover action is triggered for an index.
 You can use the {kib} Create template wizard to add the template. 
 This wizard invokes the put template API to create the template with the options you specify. 
 
-The underlying request looks like this:
+The create template request for the example template looks like this:
 
 [source,console]
 -----------------------
@@ -129,9 +128,6 @@ PUT _template/timeseries_template
 <3> The name of the alias used to reference these indices.
 Required for policies that use the rollover action.
 
-You can also invoke this API directly to add templates.
-
-
 //////////////////////////
 
 [source,console]

docs/reference/ilm/ilm-with-existing-indices.asciidoc

@@ -68,7 +68,7 @@ but a combined index needs to be retained until you are ready to delete the new
 
 . Reduce the {ilm-init} poll interval to ensure that the index doesn't 
 grow too large while waiting for the rollover check.
-By default, {ilm-init} checks rollover conditions every 10 minutes. 
+By default, {ilm-init} checks to see what actions need to be taken every 10 minutes. 
 +
 --
 [source,console]

docs/reference/ilm/index.asciidoc

@@ -5,17 +5,20 @@
 
 [partintro]
 --
-You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to
-your performance, resiliency, and retention requirements. 
+You can configure {ilm} ({ilm-init}) policies to automatically manage indices 
+according to your performance, resiliency, and retention requirements. 
 For example, you could use {ilm-init} to:
 
 * Spin up a new index when an index reaches a certain size or number of documents
 * Create a new index each day, week, or month and archive previous ones
 * Delete stale indices to enforce data retention standards
-
+
+You can create and manage index lifecycle policies through {kib} Management or the {ilm-init} APIs.
 When you enable {ilm} for {beats} or the {ls} {es} output plugin, 
-{ilm-init} is configured automatically. 
-You can modify the default policies through {kib} Management or the {ilm-init} APIs.
+default policies are configured automatically.
+
+[role="screenshot"]
+image:images/ilm/index-lifecycle-policies.png[]
 
 [TIP]
 To automatically back up your indices and manage snapshots, 
@@ -24,13 +27,15 @@ use <<getting-started-snapshot-lifecycle-management,snapshot lifecycle policies>
 * <<overview-index-lifecycle-management>>
 * <<ilm-concepts>>
 * <<getting-started-index-lifecycle-management>>
-* <<ilm-actions>>
+* <<example-using-index-lifecycle-policy>>
 * <<set-up-lifecycle-policy>>
 * <<index-lifecycle-error-handling>>
 * <<start-stop-ilm>>
 * <<ilm-with-existing-indices>>
 * <<skipping-rollover>>
 * <<index-lifecycle-and-snapshots>>
+* <<index-lifecycle-management-api>>
+* <<ilm-actions>>
 
 --
 include::ilm-overview.asciidoc[]
@@ -39,6 +44,8 @@ include::ilm-concepts.asciidoc[]
 
 include::ilm-tutorial.asciidoc[]
 
+include::example-index-lifecycle-policy.asciidoc[leveloffset=-1]
+
 include::ilm-actions.asciidoc[]
 
 include::set-up-lifecycle-policy.asciidoc[]