[DOCS] Add searchable snapshots topic. #63040

debadair · 2020-09-30T01:03:57Z

No description provided.

henningandersen

I added a couple of comments, I am in doubt about how we want to frame searchable snapshots in docs so that we make it clear that only the full copy version is available now, but can add the other variant in the future without too much confusion. I would be inclined to not introduce the fully remote variant in docs now.

docs/reference/searchable-snapshots/index.asciidoc

debadair · 2020-09-30T19:26:26Z

I added a couple of comments, I am in doubt about how we want to frame searchable snapshots in doc so that we make it clear that only the full copy version is available now, but can add the other variant in the future without too much confusion.

This initial version is something of an exploration of how we want to talk about searchable snapshots. I included the "fully-remote" option in this draft to make sure the descriptions we use now leave room for talking about it later. It introduces a number of new concepts/terms:

searchable snapshot - a snapshot of an index or data stream? that resides in a remote data store such as S3 and can be accessed dynamically at query time.
backup snapshot - a snapshot of a cluster, data stream, or index that resides in a remote data store such as S3 and is used for backup and recovery.
snapshot - captures the state of a cluster, index, or data stream at a particular point in time. See backup snapshot and searchable snapshot. (Currently defined as: A backup taken from a running Elasticsearch cluster. A snapshot can include backups of an entire cluster or only data streams and indices you specify.)
snapshot-backed index - a read-only index that relies on a searchable snapshot for redundancy.
fully-remote storage - a searchable snapshot that has no corresponding index in the cluster. Data is always loaded dynamically to to process asynchronous searches.

We might be able to come up with a better name than "fully-remote storage". I'll add the other terms to the glossary & include them in this PR so we can fine-tune the definitions.

tlrx · 2020-10-05T08:07:29Z

We might be able to come up with a better name than "fully-remote storage". I'll add the other terms to the glossary & include them in this PR so we can fine-tune the definitions.

I agree with Henning, this "fully-remote storage" is not implemented. By mentioning it I'm afraid that it would confuse users or triggers questions that we don't have the answers yet. I thinks it should be removed from the glossary and any doc.

debadair · 2020-10-15T21:16:00Z

@elasticmachine retest this please

DaveCTurner · 2020-10-16T09:47:11Z

I've expanded the conceptual docs a bit, see d4292ec. I think they cover everything I wanted to say, but please check for gaps.

I have avoided the "snapshot-backed index" terminology in favour of just calling them "searchable snapshots" or "searchable snapshot indices". I realise this is not so technically correct and we could go back on that and make a clearer distinction between the index and the snapshot behind it if we'd prefer.

henningandersen

Thanks @DaveCTurner. I left a few comments, in particular on the terminology. Otherwise looking good.

docs/reference/glossary.asciidoc

henningandersen · 2020-10-19T11:26:28Z

docs/reference/searchable-snapshots/index.asciidoc

+
+{search-snaps-cap} let you reduce your operating costs by treating the snapshot
+as the authoritative copy of some of your indices. The high reliability of the
+snapshot repository removes the need to keep multiple copies of their data in


their seems wrong, did you mean the?

I meant their (as in "belonging to the indices") but it's awkward. Reworded in 29402a5

henningandersen · 2020-10-19T11:29:18Z

docs/reference/searchable-snapshots/index.asciidoc

+
+A searchable snapshot can be searched just like any other index.
+{search-snaps-cap} are often used to access a large archive of historical data,
+for which searches may sometimes be complex and time-consuming.


I wonder if we are here framing searchable snapshots as being slow, which is not necessarily the case. I think I get that the time span of the search will be large and this adds to query time, hence async search is necessary, but I would like to move that to the meat of the section more than this introducing paragraph.

I was trying not to, but yes it still suggests slowness. I moved this to a TIP in 8beb64b and replaced this sentence with one indicating that performance should be similar to a regular index.

henningandersen · 2020-10-19T11:30:19Z

docs/reference/searchable-snapshots/index.asciidoc

+other index. You can, for instance, use <<shard-allocation-filtering>> to
+restrict these shards to a subset of your nodes.
+
+Normally you will use {search-snaps-cap} via the


"use" implies "search" in my head, I would prefer to say "create" or "mount" here?

I wanted to talk about something more general than just creating them -- after all ILM does also take care of aliases which lets you search them too. How about manage? 90a2184

henningandersen · 2020-10-19T11:33:10Z

docs/reference/searchable-snapshots/index.asciidoc

+
+If a node fails while holding some zero-replica searchable snapshot then there
+will be a brief window of time before {es} allocates these shards elsewhere.
+During this window of time the cluster health will be `red` and searches that


I believe the cluster will currently only be yellow unless we worked on this recently?

Ah yes now that the recovery source is always "snapshot" this should be the case. Hedging my bets in c43841d by saying "not green".

docs/reference/searchable-snapshots/index.asciidoc

henningandersen · 2020-10-19T11:49:13Z

docs/reference/glossary.asciidoc

@@ -450,6 +456,12 @@ in the <<glossary-mapping,mapping>>.
 // end::routing-def[]
 --

+[[glossary-searchable-snapshot]] searchable snapshot ::
+// tag::searchable-snapshot-def[]
+A <<glossary-snapshot, snapshot>> of an index or data stream that resides in a remote data store 


I wonder if the "searchable snapshot" object is the index or the snapshot? We talked about "snapshot-backed index", which is clearly the index. I think of it as the index that makes the snapshot searchable.

We could also define "searchable snapshot index/indices" here instead.

I find the current definition here slightly confusing in that any snapshot can be made searchable.

I have reworked the glossary entries in c026c9f.

docs/reference/glossary.asciidoc

jrodewig · 2020-10-20T12:52:35Z

@elasticmachine run elasticsearch-ci/docs

elasticmachine · 2020-10-20T14:32:48Z

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

elasticmachine · 2020-10-20T14:32:50Z

Pinging @elastic/es-docs (>docs)

DaveCTurner · 2020-10-20T14:34:25Z

I think this is ready for a full review now; @debadair your input would be useful but you opened the PR so I can't request a review from you.

The preview is up and running at https://elasticsearch_63040.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/searchable-snapshots.html

debadair

I think there's still some redundancy with the last section that could be cleaned up later.

docs/reference/glossary.asciidoc

debadair · 2020-10-20T22:59:30Z

docs/reference/glossary.asciidoc

@@ -450,6 +450,22 @@ in the <<glossary-mapping,mapping>>.
 // end::routing-def[]
 --

+[[glossary-searchable-snapshot]] searchable snapshot ::
+// tag::searchable-snapshot-def[]


I think we're still not quite there with these definitions. If a "searchable snapshot" is an index mounted from snapshot, do we even need the notion of the searchable snapshot index? As written, these definitions don't make the distinction between them clear.

I'd like us to be able to distinguish the index-in-the-snapshot from the index-in-the-cluster. In principle we are searching the index-in-the-snapshot, hence "searchable snapshot", and we implement this today by creating a corresponding index-in-the-cluster. I think the distinction is important since we may in future support searches directly against snapshots too. I've changed the wording slightly: "index in a snapshot" -> "snapshot of an index" -- does that help?

I think of it more like "searchable snapshot" is the concept, whereas a "searchable snapshot index" is a concrete index backed by a searchable snapshot. I.e., there is no object anywhere that is a "searchable snapshot", since all snapshots can be made searchable (through a "searchable snapshot index"). But I am also ok with the current text.

do we even need the notion of the searchable snapshot index

I think referring to an index as just a "searchable snapshot" is unintuitive, since it is an index, not a snapshot.

debadair · 2020-10-20T23:14:03Z

docs/reference/glossary.asciidoc

+An index in a <<glossary-snapshot, snapshot>> that is mounted as a
+<<glossary-searchable-snapshot-index, searchable snapshot index>> and can be
+searched as if it were a regular index.


Suggested change

An index in a <<glossary-snapshot, snapshot>> that is mounted as a

<<glossary-searchable-snapshot-index, searchable snapshot index>> and can be

searched as if it were a regular index.

A read-only index mounted from a <<glossary-snapshot, snapshot>> that can be searched like any other index. Searchable snapshots do not need

<<glossary-replica-shard,replica shards>> for resilience, since their data is

reliably stored in the snapshot repository.

debadair · 2020-10-20T23:14:25Z

docs/reference/glossary.asciidoc

+[[glossary-searchable-snapshot-index]] searchable snapshot index ::
+// tag::searchable-snapshot-index-def[]
+An <<glossary-index, index>> whose data is stored in a <<glossary-snapshot,
+snapshot>> that resides in a separate <<glossary-snapshot-repository,snapshot
+repository>> such as AWS S3. Searchable snapshot indices do not need
+<<glossary-replica-shard,replica>> shards for resilience, since their data is
+reliably stored outside the cluster.
+// end::searchable-snapshot-index-def[]


Suggested change

[[glossary-searchable-snapshot-index]] searchable snapshot index ::

// tag::searchable-snapshot-index-def[]

An <<glossary-index, index>> whose data is stored in a <<glossary-snapshot,

snapshot>> that resides in a separate <<glossary-snapshot-repository,snapshot

repository>> such as AWS S3. Searchable snapshot indices do not need

<<glossary-replica-shard,replica>> shards for resilience, since their data is

reliably stored outside the cluster.

// end::searchable-snapshot-index-def[]

docs/reference/searchable-snapshots/index.asciidoc

debadair · 2020-10-21T01:50:05Z

docs/reference/searchable-snapshots/index.asciidoc

+We recommend that you <<indices-forcemerge, force-merge>> indices to a single
+segment per shard before mounting them as {search-snaps}. Each read from a
+snapshot repository takes time and costs money, and the fewer segments there
+are the fewer reads are needed to restore the snapshot.


Moved up

Suggested change

We recommend that you <<indices-forcemerge, force-merge>> indices to a single

segment per shard before mounting them as {search-snaps}. Each read from a

snapshot repository takes time and costs money, and the fewer segments there

are the fewer reads are needed to restore the snapshot.

I'd rather this was down here. It's not very important to force-merge things before mounting them, and if you're mounting an existing snapshot you basically have no choice since you can't do anything about the segment count without restoring each index, merging it and re-snapshotting it.

docs/reference/searchable-snapshots/index.asciidoc

debadair · 2020-10-21T02:18:50Z

docs/reference/searchable-snapshots/index.asciidoc

+{search-snaps-cap} are ideal for managing a large archive of historical data.
+Historical information is typically searched less frequently than recent data
+and therefore may not need replicas for their performance benefits.
+
+You can use <<async-search>> with {search-snaps}, which is especially useful
+for more complex or time-consuming searches.


Suggested change

{search-snaps-cap} are ideal for managing a large archive of historical data.

Historical information is typically searched less frequently than recent data

and therefore may not need replicas for their performance benefits.

You can use <<async-search>> with {search-snaps}, which is especially useful

for more complex or time-consuming searches.

{search-snaps-cap} are ideal for managing large archives of historical data.

Historical information is typically searched less frequently than recent data

and performance is less important.

For more complex or time-consuming searches, you can use <<async-search>> with {search-snaps}.

I'd rather keep this wording as-is: "performance is less important" suggests to me that there's a general performance penalty for using searchable snapshots -- in fact the only drawback most of the time is the lack of replicas.

I'll apply the change to the wording re. async searches separately.

docs/reference/searchable-snapshots/index.asciidoc

Co-authored-by: debadair <[email protected]>

henningandersen

LGTM. Left a number of mostly minor comments.

henningandersen · 2020-10-22T08:33:05Z

docs/reference/searchable-snapshots/index.asciidoc

+You can control the allocation of the shards of {search-snap} indices using the
+same mechanisms as for regular indices. For example, you could use
+<<shard-allocation-filtering>> to restrict {search-snap} shards to a subset of
+your nodes.


I wonder if this should go below the ILM section in the interest of explaining the "easy/normal" option first and then the more advanced option afterwards?

I'm ambivalent -- I put it here since we're starting off by talking about how these indices are mostly maniuplated (searched & allocated) as if they were normal indices, but I've moved it in 990707b.

docs/reference/searchable-snapshots/index.asciidoc

henningandersen · 2020-10-22T09:32:52Z

docs/reference/searchable-snapshots/index.asciidoc

+[[using-searchable-snapshots]]
+=== Using {search-snaps}
+
+Searching a {search-snap} is the same as searching any other index. Search


Should this be "Searching a searchable snapshot index" to be consistent with glossary. I think it reads better too.

Same comment goes for a number of the "{search-snap}" mentions throughout.

Good point, I've added a few "index" or "shard" nouns throughout in 990707b.

henningandersen · 2020-10-22T09:38:07Z

docs/reference/glossary.asciidoc

@@ -450,6 +450,22 @@ in the <<glossary-mapping,mapping>>.
 // end::routing-def[]
 --

+[[glossary-searchable-snapshot]] searchable snapshot ::
+// tag::searchable-snapshot-def[]


I think of it more like "searchable snapshot" is the concept, whereas a "searchable snapshot index" is a concrete index backed by a searchable snapshot. I.e., there is no object anywhere that is a "searchable snapshot", since all snapshots can be made searchable (through a "searchable snapshot index"). But I am also ok with the current text.

do we even need the notion of the searchable snapshot index

I think referring to an index as just a "searchable snapshot" is unintuitive, since it is an index, not a snapshot.

* [DOCS] Add searchable snapshots topic. * [DOCS] Add definitions & remove fully-remote storage. * [DOCS] Fixed duplicate anchor. * Expand conceptual docs for searchable snapshots * Rewordings * Glossary tidy-up * Beta * Reword * More performance idea to a TIP * use -> manage * red -> not green * Missing space? * Update docs/reference/glossary.asciidoc * Fix beta label * Use more attributes, fix link titles * Apply suggestions from code review Co-authored-by: debadair <[email protected]> * Reformat * Minor rewordings * More minor rewordings * Address Henning's comments Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]>

* [DOCS] Add searchable snapshots topic. * [DOCS] Add definitions & remove fully-remote storage. * [DOCS] Fixed duplicate anchor. * Expand conceptual docs for searchable snapshots * Rewordings * Glossary tidy-up * Beta * Reword * More performance idea to a TIP * use -> manage * red -> not green * Missing space? * Update docs/reference/glossary.asciidoc * Fix beta label * Use more attributes, fix link titles * Apply suggestions from code review Co-authored-by: debadair <[email protected]> * Reformat * Minor rewordings * More minor rewordings * Address Henning's comments Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]> Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]>

* [DOCS] Add searchable snapshots topic. * [DOCS] Add definitions & remove fully-remote storage. * [DOCS] Fixed duplicate anchor. * Expand conceptual docs for searchable snapshots * Rewordings * Glossary tidy-up * Beta * Reword * More performance idea to a TIP * use -> manage * red -> not green * Missing space? * Update docs/reference/glossary.asciidoc * Fix beta label * Use more attributes, fix link titles * Apply suggestions from code review Co-authored-by: debadair <[email protected]> * Reformat * Minor rewordings * More minor rewordings * Address Henning's comments Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]>

* [DOCS] Add searchable snapshots topic. * [DOCS] Add definitions & remove fully-remote storage. * [DOCS] Fixed duplicate anchor. * Expand conceptual docs for searchable snapshots * Rewordings * Glossary tidy-up * Beta * Reword * More performance idea to a TIP * use -> manage * red -> not green * Missing space? * Update docs/reference/glossary.asciidoc * Fix beta label * Use more attributes, fix link titles * Apply suggestions from code review Co-authored-by: debadair <[email protected]> * Reformat * Minor rewordings * More minor rewordings * Address Henning's comments Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]> Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]>

* [DOCS] Add searchable snapshots topic. (#63040) * [DOCS] Add searchable snapshots topic. * [DOCS] Add definitions & remove fully-remote storage. * [DOCS] Fixed duplicate anchor. * Expand conceptual docs for searchable snapshots * Rewordings * Glossary tidy-up * Beta * Reword * More performance idea to a TIP * use -> manage * red -> not green * Missing space? * Update docs/reference/glossary.asciidoc * Fix beta label * Use more attributes, fix link titles * Apply suggestions from code review Co-authored-by: debadair <[email protected]> * Reformat * Minor rewordings * More minor rewordings * Address Henning's comments Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]> * Fixed glossary entries Co-authored-by: David Turner <[email protected]> Co-authored-by: James Rodewig <[email protected]>

debadair added >docs General docs changes :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.10.0 labels Sep 30, 2020

debadair requested review from tlrx and henningandersen September 30, 2020 01:03

henningandersen reviewed Sep 30, 2020

View reviewed changes

docs/reference/searchable-snapshots/index.asciidoc Outdated Show resolved Hide resolved

docs/reference/searchable-snapshots/index.asciidoc Outdated Show resolved Hide resolved

debadair added 3 commits October 1, 2020 17:52

[DOCS] Add searchable snapshots topic.

fef25cc

[DOCS] Add definitions & remove fully-remote storage.

b122468

[DOCS] Fixed duplicate anchor.

df1d881

debadair force-pushed the searchable-snapshots branch from bf45c7c to df1d881 Compare October 2, 2020 00:53

andreidan added v7.11.0 and removed v7.10.0 labels Oct 7, 2020

Expand conceptual docs for searchable snapshots

d4292ec

DaveCTurner requested a review from henningandersen October 16, 2020 09:47

Rewordings

003bb24

henningandersen reviewed Oct 19, 2020

View reviewed changes

DaveCTurner added 8 commits October 20, 2020 09:02

Merge branch 'master' into searchable-snapshots

b15c0e7

Glossary tidy-up

c026c9f

Beta

05b95fe

Reword

29402a5

More performance idea to a TIP

8beb64b

use -> manage

90a2184

red -> not green

c43841d

Missing space?

c1e672a

jrodewig reviewed Oct 20, 2020

View reviewed changes

docs/reference/glossary.asciidoc Outdated Show resolved Hide resolved

jrodewig and others added 2 commits October 20, 2020 08:35

Update docs/reference/glossary.asciidoc

a1e130a

Fix beta label

221dcd1

Use more attributes, fix link titles

fd9652b

DaveCTurner marked this pull request as ready for review October 20, 2020 14:32

elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Oct 20, 2020

DaveCTurner requested a review from henningandersen October 20, 2020 14:32

elasticmachine added the Team:Docs Meta label for docs team label Oct 20, 2020

debadair commented Oct 21, 2020

View reviewed changes

DaveCTurner and others added 5 commits October 22, 2020 07:59

Apply suggestions from code review

fb0ccd0

Co-authored-by: debadair <[email protected]>

Merge branch 'master' into searchable-snapshots

4929f8b

Reformat

644eccd

Minor rewordings

de75da9

More minor rewordings

26706d8

henningandersen approved these changes Oct 22, 2020

View reviewed changes

Address Henning's comments

990707b

debadair merged commit b95d9c4 into elastic:master Oct 22, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

[DOCS] Add searchable snapshots topic. #63040

[DOCS] Add searchable snapshots topic. #63040

Conversation

debadair commented Sep 30, 2020

henningandersen left a comment

Choose a reason for hiding this comment

debadair commented Sep 30, 2020 • edited Loading

tlrx commented Oct 5, 2020

debadair commented Oct 15, 2020

DaveCTurner commented Oct 16, 2020 • edited Loading

henningandersen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrodewig commented Oct 20, 2020

elasticmachine commented Oct 20, 2020

elasticmachine commented Oct 20, 2020

DaveCTurner commented Oct 20, 2020

debadair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DaveCTurner Oct 22, 2020 • edited Loading

Choose a reason for hiding this comment

henningandersen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debadair commented Sep 30, 2020 •

edited

Loading

DaveCTurner commented Oct 16, 2020 •

edited

Loading

DaveCTurner Oct 22, 2020 •

edited

Loading