Docs for translog, history retention and flushing

This commit updates the docs about translog retention and flushing to reflect
recent changes in how peer recoveries work. It also adds some docs to describe
how history is retained for replay using soft deletes and shard history
retention leases.

Relates elastic#45473

Author: DaveCTurner

docs/reference/index-modules.asciidoc

@@ -280,6 +280,10 @@ Other index settings are available in index modules:
 
     Control over the transaction log and background flush operations.
 
+<<index-modules-history-retention,History retention>>::
+
+    Control over the retention of a history of operations in the index.
+
 [float]
 [[x-pack-index-settings]]
 === [xpack]#{xpack} index settings#
@@ -305,4 +309,6 @@ include::index-modules/store.asciidoc[]
 
 include::index-modules/translog.asciidoc[]
 
+include::index-modules/history-retention.asciidoc[]
+
 include::index-modules/index-sorting.asciidoc[]

docs/reference/index-modules/history-retention.asciidoc

@@ -0,0 +1,71 @@
+[[index-modules-history-retention]]
+== History retention
+
+{es} sometimes needs to replay some of the operations that were performed on a
+shard. For instance, if a replica is briefly offline then it may be much more
+efficient to replay the few operations it missed while it was offline than to
+rebuild it from scratch. Also, {ccr} works by performing operations on the
+leader cluster and then replaying those operations on the follower cluster.
+
+At the Lucene level there are really only two write operations that {es}
+performs on an index: a new document may be indexed, or an existing document may
+be deleted. Updates are implemented as an atomic operation comprising the
+deletion of the old document followed by the indexing of the new document. A
+document indexed into Lucene already contains all the information needed to
+replay that indexing operation, but this is not true of document deletions. To
+solve this, {es} uses a feature called _soft deletes_ to preserve recent
+deletions in the Lucene index so that they can be replayed.
+
+It is important that {es} eventually discards any soft-deleted documents to
+prevent long-running indices from growing without bound. {es} tries not to
+discard any soft-deleted documents that it expects to need in the future,
+because if it needs to replay a discarded operation then it has no choice but to
+perform a full copy of the whole index to ensure that everything remains
+correctly synchronized. Copying the whole index may take considerable time and
+resources, which is why {es} tries to avoid this where possible.
+
+{es} keeps track of the operations it expects to need to replay in future using
+a mechanism called _shard history retention leases_. Each process that might
+need operations to be replayed must first takes out a shard history retention
+lease. For example, this process might be a replica of a shard or it might be a
+shard of a follower index when using {ccr}. Each retention lease keeps track of
+the sequence number of the first operation that the process has not received.
+As the process receives operations, it increases the sequence number contained
+in its retention lease to indicate that it will not need to replay those
+operations in future. {es} can discard soft-deleted operations once they are not
+being held by any retention lease.
+
+If a process crashes then it cannot update its retention lease any more, which
+means that {es} will preserve any new operations so they can be replayed when
+the crashed process recovers. However, retention leases only last for a limited
+amount of time. If the process does not recover quickly enough then its
+retention lease may expire. This protects {es} from retaining history forever if
+a process crashes permanently, because once a retention lease has expired {es}
+can start to discard history again. If a process recovers after its retention
+lease has expired then {es} will fall back to copying the whole index since it
+can no longer simply replay the missing history. The expiry time of a retention
+lease defaults to `12h`.
+
+Soft deletes are enabled by default on indices created in recent versions, but
+they can be explicitly enabled or disabled at index creation time. If soft
+deletes are disabled then peer recoveries can still sometimes take place by
+copying just the missing operations from the translog
+<<index-modules-translog-retention,as long as those operations are retained
+there>>. {ccr-cap} will not function if soft deletes are disabled.
+
+[float]
+=== History retention settings
+
+`index.soft_deletes.enabled`::
+
+  Whether or not soft deletes are enabled on the index. Soft deletes can only be
+  configured at index creation and only on indices created on or after 6.5.0.
+  The default value is `true`.
+
+`index.soft_deletes.retention_lease.period`::
+
+  The maximum period to retain a shard history retention lease before it is
+  considered expired. Shard history retention leases ensure that soft deletes
+  are retained during merges on the Lucene index. If a soft delete is merged
+  away before it can be replicated to a follower the following process will fail
+  due to incomplete history on the leader.  The default value is `12h`.

docs/reference/index-modules/translog.asciidoc

@@ -9,53 +9,55 @@ failure.
 
 Because Lucene commits are too expensive to perform on every individual change,
 each shard copy also has a _transaction log_ known as its _translog_ associated
-with it. All index and delete operations are written to the translog after
-being processed by the internal Lucene index but before they are acknowledged.
-In the event of a crash, recent transactions that have been acknowledged but
-not yet included in the last Lucene commit can instead be recovered from the
-translog when the shard recovers.
-
-An Elasticsearch flush is the process of performing a Lucene commit and
-starting a new translog. Flushes are performed automatically in the background
-in order to make sure the translog doesn't grow too large, which would make
-replaying its operations take a considerable amount of time during recovery.
-The ability to perform a flush manually is also exposed through an API,
-although this is rarely needed.
+with it. All index and delete operations are written to the translog after being
+processed by the internal Lucene index but before they are acknowledged. In the
+event of a crash, recent operations that have been acknowledged but not yet
+included in the last Lucene commit are instead recovered from the translog when
+the shard recovers.
+
+An {es} <<indices-flush,flush>> is the process of performing a Lucene commit and
+starting a new translog generation. Flushes are performed automatically in the
+background in order to make sure the translog does not grow too large, which
+would make replaying its operations take a considerable amount of time during
+recovery. The ability to perform a flush manually is also exposed through an
+API, although this is rarely needed.
 
 [float]
 === Translog settings
 
 The data in the translog is only persisted to disk when the translog is
-++fsync++ed and committed.  In the event of a hardware failure or an operating
+++fsync++ed and committed. In the event of a hardware failure or an operating
 system crash or a JVM crash or a shard failure, any data written since the
 previous translog commit will be lost.
 
-By default, `index.translog.durability` is set to `request` meaning that Elasticsearch will only report success of an index, delete,
-update, or bulk request to the client after the translog has been successfully
-++fsync++ed and committed on the primary and on every allocated replica. If
-`index.translog.durability` is set to `async` then Elasticsearch ++fsync++s
-and commits the translog every `index.translog.sync_interval` (defaults to 5 seconds).
+By default, `index.translog.durability` is set to `request` meaning that
+Elasticsearch will only report success of an index, delete, update, or bulk
+request to the client after the translog has been successfully ++fsync++ed and
+committed on the primary and on every allocated replica. If
+`index.translog.durability` is set to `async` then Elasticsearch ++fsync++s and
+commits the translog only every `index.translog.sync_interval` which means that
+any operations that were performed just before a crash may be lost when the node
+recovers.
 
 The following <<indices-update-settings,dynamically updatable>> per-index
 settings control the behaviour of the translog:
 
 `index.translog.sync_interval`::
 
-How often the translog is ++fsync++ed to disk and committed, regardless of
-write operations. Defaults to `5s`. Values less than `100ms` are not allowed.
+  How often the translog is ++fsync++ed to disk and committed, regardless of
+  write operations. Defaults to `5s`. Values less than `100ms` are not allowed.
 
 `index.translog.durability`::
 +
 --
 
 Whether or not to `fsync` and commit the translog after every index, delete,
-update, or bulk request.  This setting accepts the following parameters:
+update, or bulk request. This setting accepts the following parameters:
 
 `request`::
 
-    (default) `fsync` and commit after every request. In the event
-    of hardware failure, all acknowledged writes will already have been
-    committed to disk.
+    (default) `fsync` and commit after every request. In the event of hardware
+    failure, all acknowledged writes will already have been committed to disk.
 
 `async`::
 
@@ -66,33 +68,43 @@ update, or bulk request.  This setting accepts the following parameters:
 
 `index.translog.flush_threshold_size`::
 
-The translog stores all operations that are not yet safely persisted in Lucene
-(i.e., are not part of a Lucene commit point). Although these operations are
-available for reads, they will need to be reindexed if the shard was to
-shutdown and has to be recovered. This settings controls the maximum total size
-of these operations, to prevent recoveries from taking too long. Once the
-maximum size has been reached a flush will happen, generating a new Lucene
-commit point. Defaults to `512mb`.
+  The translog stores all operations that are not yet safely persisted in Lucene
+  (i.e., are not part of a Lucene commit point). Although these operations are
+  available for reads, they will need to be replayed if the shard was stopped
+  and had to be recovered. This setting controls the maximum total size of these
+  operations, to prevent recoveries from taking too long. Once the maximum size
+  has been reached a flush will happen, generating a new Lucene commit point.
+  Defaults to `512mb`.
 
-`index.translog.retention.size`::
-
-When soft deletes is disabled (enabled by default in 7.0 or later),
-`index.translog.retention.size` controls the total size of translog files to keep.
-Keeping more translog files increases the chance of performing an operation based
-sync when recovering replicas. If the translog files are not sufficient,
-replica recovery will fall back to a file based sync. Defaults to `512mb`
+[float]
+[[index-modules-translog-retention]]
+==== Translog retention
+
+If an index is not using <<index-modules-history-retention,soft deletes>> to
+retain historical operations then {es} recovers each replica shard by replaying
+operations from the primary's translog. This means it is important for the
+primary to preserve extra operations in its translog in case it needs to
+rebuild a replica. Moreover it is important for each replica to preserve extra
+operations in its translog in case it is promoted to primary and then needs to
+rebuild its own replicas in turn. The following settings control how much
+translog is retained for peer recoveries.
 
-Both `index.translog.retention.size` and `index.translog.retention.age` should not
-be specified unless soft deletes is disabled as they will be ignored.
+`index.translog.retention.size`::
 
+  This controls the total size of translog files to keep for each shard.
+  Keeping more translog files increases the chance of performing an operation
+  based sync when recovering a replica. If the translog files are not
+  sufficient, replica recovery will fall back to a file based sync. Defaults to
+  `512mb`. This setting is ignored, and should not be set, if soft deletes are
+  enabled. Soft deletes are enabled by default in indices created in {es}
+  versions 7.0.0 and later.
 
 `index.translog.retention.age`::
 
-When soft deletes is disabled (enabled by default in 7.0 or later),
-`index.translog.retention.age` controls the maximum duration for which translog
-files to keep. Keeping more translog files increases the chance of performing an
-operation based sync when recovering replicas. If the translog files are not sufficient,
-replica recovery will fall back to a file based sync. Defaults to `12h`
-
-Both `index.translog.retention.size` and `index.translog.retention.age` should not
-be specified unless soft deletes is disabled as they will be ignored.
+  This controls the maximum duration for which translog files are kept by each
+  shard. Keeping more translog files increases the chance of performing an
+  operation based sync when recovering replicas. If the translog files are not
+  sufficient, replica recovery will fall back to a file based sync. Defaults to
+  `12h`. This setting is ignored, and should not be set, if soft deletes are
+  enabled. Soft deletes are enabled by default in indices created in {es}
+  versions 7.0.0 and later.