Skip to content

[DOCS] Add how-to guide for time series data #71195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Apr 5, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions docs/reference/data-streams/set-up-a-data-stream.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ PUT _ilm/policy/my-lifecycle-policy
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "50gb"
}
}
Expand All @@ -59,15 +58,15 @@ PUT _ilm/policy/my-lifecycle-policy
"min_age": "60d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my-snapshot-repo"
"snapshot_repository": "found-snapshots"
}
}
},
"frozen": {
"min_age": "90d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my-snapshot-repo"
"snapshot_repository": "found-snapshots"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,10 @@ To get the most out of your time series data in {es}, follow these steps:
tiers>> to automatically move older data to nodes with less expensive hardware
as it ages. This helps improve performance and reduce storage costs.

We recommend setting up at least a hot tier for your cluster. Add warm, cold,
and frozen tiers based on your performance and storage needs.

Use high performance nodes in the hot and warm tiers for faster indexing and
faster searches on your most recent data. Use slower, less expensive nodes in
the cold and frozen tiers to reduce costs.
The hot tier is required. The warm, cold, and frozen tiers are optional. Use
high performance nodes in the hot and warm tiers for faster indexing and faster
searches on your most recent data. Use slower, less expensive nodes in the cold
and frozen tiers to reduce costs.

The steps for setting up data tiers vary based on your deployment type:

Expand All @@ -40,8 +38,8 @@ include::{es-repo-dir}/tab-widgets/data-tiers-widget.asciidoc[]
[[register-snapshot-repository]]
=== Step 2. Register a snapshot repository

The cold and frozen tiers use <<searchable-snapshots,searchable snapshots>> to
reduce local storage costs.
The cold and frozen tiers can use <<searchable-snapshots,searchable snapshots>>
to reduce local storage costs.

To create snapshots, you must register a snapshot repository. The repository is
typically a remote object store, such as Google Cloud Storage (GCS) or Azure
Expand All @@ -60,17 +58,13 @@ A <<data-streams,data stream>> stores your data across multiple backing
indices. {ilm-init} uses an <<ilm-index-lifecycle,index lifecycle policy>> to
automatically move these indices through your data tiers.

A lifecycle policy contains phases that correspond to data tiers. You can
specify automated actions to take on indices in each phase. A policy can also
include a delete phase to safely delete indices you no longer need.

If you use {fleet} or {agent}, edit one of {es}'s built-in lifecycle policies.
If you use a custom application, create your own policy. In either case,
ensure your policy:

* Includes a phase for each data tier you've configured.
* Calculates the threshold, or `min_age`, for phase transition from rollover.
* Uses searchable snapshots in the cold and frozen phases, if included.
* Uses searchable snapshots in the cold and frozen phases, if desired.
* Includes a delete phase, if needed.

include::{es-repo-dir}/tab-widgets/ilm-widget.asciidoc[]
Expand Down
5 changes: 2 additions & 3 deletions docs/reference/tab-widgets/ilm.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ PUT _ilm/policy/logs
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "50gb"
}
}
Expand All @@ -42,15 +41,15 @@ PUT _ilm/policy/logs
"min_age": "60d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my-snapshot-repo"
"snapshot_repository": "found-snapshots"
}
}
},
"frozen": {
"min_age": "90d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my-snapshot-repo"
"snapshot_repository": "found-snapshots"
}
}
},
Expand Down