-
Notifications
You must be signed in to change notification settings - Fork 25.2k
[DOCS] Add searchable snapshots topic. #63040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
fef25cc
[DOCS] Add searchable snapshots topic.
debadair b122468
[DOCS] Add definitions & remove fully-remote storage.
debadair df1d881
[DOCS] Fixed duplicate anchor.
debadair d4292ec
Expand conceptual docs for searchable snapshots
DaveCTurner 003bb24
Rewordings
DaveCTurner b15c0e7
Merge branch 'master' into searchable-snapshots
DaveCTurner c026c9f
Glossary tidy-up
DaveCTurner 05b95fe
Beta
DaveCTurner 29402a5
Reword
DaveCTurner 8beb64b
More performance idea to a TIP
DaveCTurner 90a2184
use -> manage
DaveCTurner c43841d
red -> not green
DaveCTurner c1e672a
Missing space?
DaveCTurner a1e130a
Update docs/reference/glossary.asciidoc
jrodewig 221dcd1
Fix beta label
DaveCTurner fd9652b
Use more attributes, fix link titles
DaveCTurner fb0ccd0
Apply suggestions from code review
DaveCTurner 4929f8b
Merge branch 'master' into searchable-snapshots
DaveCTurner 644eccd
Reformat
DaveCTurner de75da9
Minor rewordings
DaveCTurner 26706d8
More minor rewordings
DaveCTurner 990707b
Address Henning's comments
DaveCTurner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -450,6 +450,22 @@ in the <<glossary-mapping,mapping>>. | |||||||||||||||||
// end::routing-def[] | ||||||||||||||||||
-- | ||||||||||||||||||
|
||||||||||||||||||
[[glossary-searchable-snapshot]] searchable snapshot :: | ||||||||||||||||||
// tag::searchable-snapshot-def[] | ||||||||||||||||||
A <<glossary-snapshot, snapshot>> of an index that has been mounted as a | ||||||||||||||||||
<<glossary-searchable-snapshot-index, searchable snapshot index>> and can be | ||||||||||||||||||
searched as if it were a regular index. | ||||||||||||||||||
// end::searchable-snapshot-def[] | ||||||||||||||||||
|
||||||||||||||||||
[[glossary-searchable-snapshot-index]] searchable snapshot index :: | ||||||||||||||||||
// tag::searchable-snapshot-index-def[] | ||||||||||||||||||
An <<glossary-index, index>> whose data is stored in a <<glossary-snapshot, | ||||||||||||||||||
snapshot>> that resides in a separate <<glossary-snapshot-repository,snapshot | ||||||||||||||||||
repository>> such as AWS S3. Searchable snapshot indices do not need | ||||||||||||||||||
<<glossary-replica-shard,replica>> shards for resilience, since their data is | ||||||||||||||||||
reliably stored outside the cluster. | ||||||||||||||||||
// end::searchable-snapshot-index-def[] | ||||||||||||||||||
Comment on lines
+460
to
+467
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
[[glossary-shard]] shard :: | ||||||||||||||||||
+ | ||||||||||||||||||
-- | ||||||||||||||||||
|
@@ -486,9 +502,11 @@ See the {ref}/indices-shrink-index.html[shrink index API]. | |||||||||||||||||
|
||||||||||||||||||
[[glossary-snapshot]] snapshot :: | ||||||||||||||||||
// tag::snapshot-def[] | ||||||||||||||||||
A backup taken from a running {es} cluster. | ||||||||||||||||||
A snapshot can include backups of an entire cluster or only data streams and | ||||||||||||||||||
indices you specify. | ||||||||||||||||||
Captures the state of the whole cluster or of particular indices or data | ||||||||||||||||||
streams at a particular point in time. Snapshots provide a back up of a running | ||||||||||||||||||
cluster, ensuring you can restore your data in the event of a failure. You can | ||||||||||||||||||
also mount indices or datastreams from snapshots as read-only | ||||||||||||||||||
{ref}/glossary.html#glossary-searchable-snapshot-index[searchable snapshots]. | ||||||||||||||||||
// end::snapshot-def[] | ||||||||||||||||||
|
||||||||||||||||||
[[glossary-snapshot-lifecycle-policy]] snapshot lifecycle policy :: | ||||||||||||||||||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
[[searchable-snapshots]] | ||
== {search-snaps-cap} | ||
|
||
DaveCTurner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
beta::[] | ||
|
||
DaveCTurner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
{search-snaps-cap} let you reduce your operating costs by using | ||
<<snapshot-restore, snapshots>> for resiliency rather than maintaining | ||
<<scalability,replica shards>> within a cluster. When you mount an index from a | ||
snapshot as a {search-snap}, {es} copies the index shards to local storage | ||
within the cluster. This ensures that search performance is comparable to | ||
searching any other index, and minimizes the need to access the snapshot | ||
repository. Should a node fail, shards of a {search-snap} index are | ||
automatically recovered from the snapshot repository. | ||
|
||
This can result in significant cost savings. With {search-snaps}, you may be | ||
able to halve your cluster size without increasing the risk of data loss or | ||
reducing the amount of data you can search. Because {search-snaps} rely on the | ||
same snapshot mechanism you use for backups, they have a minimal impact on your | ||
snapshot repository storage costs. | ||
|
||
[discrete] | ||
[[using-searchable-snapshots]] | ||
=== Using {search-snaps} | ||
DaveCTurner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Searching a {search-snap} index is the same as searching any other index. | ||
Search performance is comparable to regular indices because the shard data is | ||
copied onto nodes in the cluster when the {search-snap} is mounted. | ||
|
||
By default, {search-snap} indices have no replicas. The underlying snapshot | ||
provides resilience and the query volume is expected to be low enough that a | ||
single shard copy will be sufficient. However, if you need to support a higher | ||
query volume, you can add replicas by adjusting the `index.number_of_replicas` | ||
index setting. | ||
|
||
If a node fails and {search-snap} shards need to be restored from the snapshot, | ||
there is a brief window of time while {es} allocates the shards to other nodes | ||
where the cluster health will not be `green`. Searches that hit these shards | ||
will fail or return partial results until they are reallocated. | ||
|
||
You typically manage {search-snaps} through {ilm-init}. The | ||
<<ilm-searchable-snapshot, searchable snapshots>> action automatically converts | ||
an index to a {search-snap} when it reaches the `cold` phase. You can also make | ||
indices in existing snapshots searchable by manually mounting them as | ||
{search-snaps} with the <<searchable-snapshots-api-mount-snapshot, mount | ||
snapshot>> API. | ||
|
||
To mount an index from a snapshot that contains multiple indices, we recommend | ||
creating a <<clone-snapshot-api, clone>> of the snapshot that contains only the | ||
index you want to search, and mounting the clone. You cannot delete a snapshot | ||
if it has any mounted indices, so creating a clone enables you to manage the | ||
lifecycle of the backup snapshot independently of any {search-snaps}. | ||
|
||
You can control the allocation of the shards of {search-snap} indices using the | ||
same mechanisms as for regular indices. For example, you could use | ||
<<shard-allocation-filtering>> to restrict {search-snap} shards to a subset of | ||
your nodes. | ||
|
||
We recommend that you <<indices-forcemerge, force-merge>> indices to a single | ||
segment per shard before taking a snapshot that will be mounted as a | ||
{search-snap} index. Each read from a snapshot repository takes time and costs | ||
money, and the fewer segments there are the fewer reads are needed to restore | ||
the snapshot. | ||
|
||
[TIP] | ||
==== | ||
{search-snaps-cap} are ideal for managing a large archive of historical data. | ||
Historical information is typically searched less frequently than recent data | ||
and therefore may not need replicas for their performance benefits. | ||
|
||
For more complex or time-consuming searches, you can use <<async-search>> with | ||
{search-snaps}. | ||
==== | ||
|
||
[discrete] | ||
[[how-searchable-snapshots-work]] | ||
=== How {search-snaps} work | ||
|
||
When an index is mounted from a snapshot, {es} allocates its shards to data | ||
nodes within the cluster. The data nodes then automatically restore the shard | ||
data from the repository onto local storage. Once the restore process | ||
completes, these shards respond to searches using the data held in local | ||
storage and do not need to access the repository. This avoids incurring the | ||
cost or performance penalty associated with reading data from the repository. | ||
|
||
If a node holding one of these shards fails, {es} automatically allocates it to | ||
another node, and that node restores the shard data from the repository. No | ||
replicas are needed, and no complicated monitoring or orchestration is | ||
necessary to restore lost shards. | ||
|
||
{es} restores {search-snap} shards in the background and you can search them | ||
even if they have not been fully restored. If a search hits a {search-snap} | ||
shard before it has been fully restored, {es} eagerly retrieves the data needed | ||
for the search. If a shard is freshly allocated to a node and still warming up, | ||
some searches will be slower. However, searches typically access a very small | ||
fraction of the total shard data so the performance penalty is typically small. | ||
|
||
Replicas of {search-snaps} shards are restored by copying data from the | ||
snapshot repository. In contrast, replicas of regular indices are restored by | ||
copying data from the primary. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we're still not quite there with these definitions. If a "searchable snapshot" is an index mounted from snapshot, do we even need the notion of the searchable snapshot index? As written, these definitions don't make the distinction between them clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like us to be able to distinguish the index-in-the-snapshot from the index-in-the-cluster. In principle we are searching the index-in-the-snapshot, hence "searchable snapshot", and we implement this today by creating a corresponding index-in-the-cluster. I think the distinction is important since we may in future support searches directly against snapshots too. I've changed the wording slightly: "index in a snapshot" -> "snapshot of an index" -- does that help?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think of it more like "searchable snapshot" is the concept, whereas a "searchable snapshot index" is a concrete index backed by a searchable snapshot. I.e., there is no object anywhere that is a "searchable snapshot", since all snapshots can be made searchable (through a "searchable snapshot index"). But I am also ok with the current text.
I think referring to an index as just a "searchable snapshot" is unintuitive, since it is an index, not a snapshot.