Docs for translog, history retention and flushing (#46245)

This commit updates the docs about translog retention and flushing to reflect
recent changes in how peer recoveries work. It also adds some docs to describe
how history is retained for replay using soft deletes and shard history
retention leases.

Relates #45473

Author: DaveCTurner

docs/reference/index-modules.asciidoc

@@ -280,6 +280,10 @@ Other index settings are available in index modules:
 
     Control over the transaction log and background flush operations.
 
+<<index-modules-history-retention,History retention>>::
+
+    Control over the retention of a history of operations in the index.
+
 [float]
 [[x-pack-index-settings]]
 === [xpack]#{xpack} index settings#
@@ -305,4 +309,6 @@ include::index-modules/store.asciidoc[]
 
 include::index-modules/translog.asciidoc[]
 
+include::index-modules/history-retention.asciidoc[]
+
 include::index-modules/index-sorting.asciidoc[]

docs/reference/index-modules/history-retention.asciidoc

@@ -0,0 +1,72 @@
+[[index-modules-history-retention]]
+== History retention
+
+{es} sometimes needs to replay some of the operations that were performed on a
+shard. For instance, if a replica is briefly offline then it may be much more
+efficient to replay the few operations it missed while it was offline than to
+rebuild it from scratch. Similarly, {ccr} works by performing operations on the
+leader cluster and then replaying those operations on the follower cluster.
+
+At the Lucene level there are really only two write operations that {es}
+performs on an index: a new document may be indexed, or an existing document may
+be deleted. Updates are implemented by atomically deleting the old document and
+then indexing the new document. A document indexed into Lucene already contains
+all the information needed to replay that indexing operation, but this is not
+true of document deletions. To solve this, {es} uses a feature called _soft
+deletes_ to preserve recent deletions in the Lucene index so that they can be
+replayed.
+
+{es} only preserves certain recently-deleted documents in the index because a
+soft-deleted document still takes up some space. Eventually {es} will fully
+discard these soft-deleted documents to free up that space so that the index
+does not grow larger and larger over time. Fortunately {es} does not need to be
+able to replay every operation that has ever been performed on a shard, because
+it is always possible to make a full copy of a shard on a remote node. However,
+copying the whole shard may take much longer than replaying a few missing
+operations, so {es} tries to retain all of the operations it expects to need to
+replay in future.
+
+{es} keeps track of the operations it expects to need to replay in future using
+a mechanism called _shard history retention leases_. Each shard copy that might
+need operations to be replayed must first create a shard history retention lease
+for itself. For example, this shard copy might be a replica of a shard or it
+might be a shard of a follower index when using {ccr}. Each retention lease
+keeps track of the sequence number of the first operation that the corresponding
+shard copy has not received. As the shard copy receives new operations, it
+increases the sequence number contained in its retention lease to indicate that
+it will not need to replay those operations in future. {es} discards
+soft-deleted operations once they are not being held by any retention lease.
+
+If a shard copy fails then it stops updating its shard history retention lease,
+which means that {es} will preserve all new operations so they can be replayed
+when the failed shard copy recovers. However, retention leases only last for a
+limited amount of time. If the shard copy does not recover quickly enough then
+its retention lease may expire. This protects {es} from retaining history
+forever if a shard copy fails permanently, because once a retention lease has
+expired {es} can start to discard history again. If a shard copy recovers after
+its retention lease has expired then {es} will fall back to copying the whole
+index since it can no longer simply replay the missing history. The expiry time
+of a retention lease defaults to `12h` which should be long enough for most
+reasonable recovery scenarios.
+
+Soft deletes are enabled by default on indices created in recent versions, but
+they can be explicitly enabled or disabled at index creation time. If soft
+deletes are disabled then peer recoveries can still sometimes take place by
+copying just the missing operations from the translog
+<<index-modules-translog-retention,as long as those operations are retained
+there>>. {ccr-cap} will not function if soft deletes are disabled.
+
+[float]
+=== History retention settings
+
+`index.soft_deletes.enabled`::
+
+  Whether or not soft deletes are enabled on the index. Soft deletes can only be
+  configured at index creation and only on indices created on or after 6.5.0.
+  The default value is `true`.
+
+`index.soft_deletes.retention_lease.period`::
+
+  The maximum length of time to retain a shard history retention lease before
+  it expires and the history that it retains can be discarded. The default
+  value is `12h`.

docs/reference/index-modules/translog.asciidoc

@@ -7,55 +7,57 @@ delete operation. Changes that happen after one commit and before another will
 be removed from the index by Lucene in the event of process exit or hardware
 failure.
 
-Because Lucene commits are too expensive to perform on every individual change,
-each shard copy also has a _transaction log_ known as its _translog_ associated
-with it. All index and delete operations are written to the translog after
+Lucene commits are too expensive to perform on every individual change, so each
+shard copy also writes operations into its _transaction log_ known as the
+_translog_. All index and delete operations are written to the translog after
 being processed by the internal Lucene index but before they are acknowledged.
-In the event of a crash, recent transactions that have been acknowledged but
-not yet included in the last Lucene commit can instead be recovered from the
-translog when the shard recovers.
+In the event of a crash, recent operations that have been acknowledged but not
+yet included in the last Lucene commit are instead recovered from the translog
+when the shard recovers.
 
-An Elasticsearch flush is the process of performing a Lucene commit and
-starting a new translog. Flushes are performed automatically in the background
-in order to make sure the translog doesn't grow too large, which would make
-replaying its operations take a considerable amount of time during recovery.
-The ability to perform a flush manually is also exposed through an API,
-although this is rarely needed.
+An {es} <<indices-flush,flush>> is the process of performing a Lucene commit and
+starting a new translog generation. Flushes are performed automatically in the
+background in order to make sure the translog does not grow too large, which
+would make replaying its operations take a considerable amount of time during
+recovery. The ability to perform a flush manually is also exposed through an
+API, although this is rarely needed.
 
 [float]
 === Translog settings
 
 The data in the translog is only persisted to disk when the translog is
-++fsync++ed and committed.  In the event of a hardware failure or an operating
+++fsync++ed and committed. In the event of a hardware failure or an operating
 system crash or a JVM crash or a shard failure, any data written since the
 previous translog commit will be lost.
 
-By default, `index.translog.durability` is set to `request` meaning that Elasticsearch will only report success of an index, delete,
-update, or bulk request to the client after the translog has been successfully
-++fsync++ed and committed on the primary and on every allocated replica. If
-`index.translog.durability` is set to `async` then Elasticsearch ++fsync++s
-and commits the translog every `index.translog.sync_interval` (defaults to 5 seconds).
+By default, `index.translog.durability` is set to `request` meaning that
+Elasticsearch will only report success of an index, delete, update, or bulk
+request to the client after the translog has been successfully ++fsync++ed and
+committed on the primary and on every allocated replica. If
+`index.translog.durability` is set to `async` then Elasticsearch ++fsync++s and
+commits the translog only every `index.translog.sync_interval` which means that
+any operations that were performed just before a crash may be lost when the node
+recovers.
 
 The following <<indices-update-settings,dynamically updatable>> per-index
 settings control the behaviour of the translog:
 
 `index.translog.sync_interval`::
 
-How often the translog is ++fsync++ed to disk and committed, regardless of
-write operations. Defaults to `5s`. Values less than `100ms` are not allowed.
+  How often the translog is ++fsync++ed to disk and committed, regardless of
+  write operations. Defaults to `5s`. Values less than `100ms` are not allowed.
 
 `index.translog.durability`::
 +
 --
 
 Whether or not to `fsync` and commit the translog after every index, delete,
-update, or bulk request.  This setting accepts the following parameters:
+update, or bulk request. This setting accepts the following parameters:
 
 `request`::
 
-    (default) `fsync` and commit after every request. In the event
-    of hardware failure, all acknowledged writes will already have been
-    committed to disk.
+    (default) `fsync` and commit after every request. In the event of hardware
+    failure, all acknowledged writes will already have been committed to disk.
 
 `async`::
 
@@ -66,33 +68,43 @@ update, or bulk request.  This setting accepts the following parameters:
 
 `index.translog.flush_threshold_size`::
 
-The translog stores all operations that are not yet safely persisted in Lucene
-(i.e., are not part of a Lucene commit point). Although these operations are
-available for reads, they will need to be reindexed if the shard was to
-shutdown and has to be recovered. This settings controls the maximum total size
-of these operations, to prevent recoveries from taking too long. Once the
-maximum size has been reached a flush will happen, generating a new Lucene
-commit point. Defaults to `512mb`.
+  The translog stores all operations that are not yet safely persisted in Lucene
+  (i.e., are not part of a Lucene commit point). Although these operations are
+  available for reads, they will need to be replayed if the shard was stopped
+  and had to be recovered. This setting controls the maximum total size of these
+  operations, to prevent recoveries from taking too long. Once the maximum size
+  has been reached a flush will happen, generating a new Lucene commit point.
+  Defaults to `512mb`.
 
-`index.translog.retention.size`::
-
-When soft deletes is disabled (enabled by default in 7.0 or later),
-`index.translog.retention.size` controls the total size of translog files to keep.
-Keeping more translog files increases the chance of performing an operation based
-sync when recovering replicas. If the translog files are not sufficient,
-replica recovery will fall back to a file based sync. Defaults to `512mb`
+[float]
+[[index-modules-translog-retention]]
+==== Translog retention
+
+If an index is not using <<index-modules-history-retention,soft deletes>> to
+retain historical operations then {es} recovers each replica shard by replaying
+operations from the primary's translog. This means it is important for the
+primary to preserve extra operations in its translog in case it needs to
+rebuild a replica. Moreover it is important for each replica to preserve extra
+operations in its translog in case it is promoted to primary and then needs to
+rebuild its own replicas in turn. The following settings control how much
+translog is retained for peer recoveries.
 
-Both `index.translog.retention.size` and `index.translog.retention.age` should not
-be specified unless soft deletes is disabled as they will be ignored.
+`index.translog.retention.size`::
 
+  This controls the total size of translog files to keep for each shard.
+  Keeping more translog files increases the chance of performing an operation
+  based sync when recovering a replica. If the translog files are not
+  sufficient, replica recovery will fall back to a file based sync. Defaults to
+  `512mb`. This setting is ignored, and should not be set, if soft deletes are
+  enabled. Soft deletes are enabled by default in indices created in {es}
+  versions 7.0.0 and later.
 
 `index.translog.retention.age`::
 
-When soft deletes is disabled (enabled by default in 7.0 or later),
-`index.translog.retention.age` controls the maximum duration for which translog
-files to keep. Keeping more translog files increases the chance of performing an
-operation based sync when recovering replicas. If the translog files are not sufficient,
-replica recovery will fall back to a file based sync. Defaults to `12h`
-
-Both `index.translog.retention.size` and `index.translog.retention.age` should not
-be specified unless soft deletes is disabled as they will be ignored.
+  This controls the maximum duration for which translog files are kept by each
+  shard. Keeping more translog files increases the chance of performing an
+  operation based sync when recovering replicas. If the translog files are not
+  sufficient, replica recovery will fall back to a file based sync. Defaults to
+  `12h`. This setting is ignored, and should not be set, if soft deletes are
+  enabled. Soft deletes are enabled by default in indices created in {es}
+  versions 7.0.0 and later.