[DOCS] Add docs for restoring to new cluster (#79683) (#79875)

When restoring a snapshot to a new cluster, users may expect the cluster
to not contain any conflicting indices or data streams. However, some
features, such as the GeoIP processor, automatically create indices at
startup.

This adds and updates related procedures in the restore a snapshot tutorial.
I plan to improve other documentation related to feature states in snapshots
in a separate PR(s).

This PR also updates the restore snapshot API's example to include
the `indices` and `feature_states` parameters.

Relates to #79675

Author: jrodewig

docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc

@@ -53,8 +53,14 @@ POST /index_4/_close?wait_for_active_shards=0
 [source,console]
 ----
 POST /_snapshot/my_repository/my_snapshot/_restore
+{
+  "indices": "*,-.*",
+  "feature_states": [ "geoip" ]
+}
 ----
 // TEST[s/_restore/_restore?wait_for_completion=true/]
+// TEST[s/",/"/]
+// TEST[s/"feature_states": \[ "geoip" \]//]
 
 [[restore-snapshot-api-request]]
 ==== {api-request-title}

docs/reference/snapshot-restore/restore-snapshot.asciidoc

@@ -10,6 +10,7 @@ In this guide, you'll learn how to:
 
 * Get a list of available snapshots
 * Restore an index or data stream from a snapshot
+* Restore a feature state
 * Restore an entire cluster
 * Monitor the restore operation
 * Cancel an ongoing restore
@@ -33,7 +34,7 @@ When restoring data from a snapshot, keep the following in mind:
 * If you restore a data stream, you also restore its backing indices.
 
 * You can only restore an existing index if it's <<indices-close,closed>> and
-the index in the snapshot has the same number of primary shards. 
+the index in the snapshot has the same number of primary shards.
 
 * You can't restore an existing open index. This includes
 backing indices for a data stream.
@@ -79,15 +80,21 @@ GET _snapshot/my_repository/*?verbose=false
 You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the
 <<restore-snapshot-api,restore snapshot API>>.
 
-In most cases, you only need to restore a specific index or data stream from a
-snapshot. However, you can't restore an existing open index.
+By default, a restore request attempts to restore all indices and data streams
+in a snapshot, including <<system-indices,system indices and system data
+streams>>. In most cases, you only need to restore a specific index or data
+stream from a snapshot. However, you can't restore an existing open index.
 
-To avoid conflicts with existing indices and data streams, use one of the
-following methods:
+If you're restoring data to a pre-existing cluster, use one of the
+following methods to avoid conflicts with existing indices and data streams:
 
 * <<delete-restore>>
 * <<rename-on-restore>>
 
+Some {es} features automatically create system indices on cluster startup. To
+avoid conflicts with these system indices when restoring data to a new cluster,
+see <<restore-exclude-system-indices>>.
+
 [discrete]
 [[delete-restore]]
 ==== Delete and restore
@@ -113,18 +120,12 @@ DELETE _data_stream/logs-my_app-default
 ----
 // TEST[setup:setup-snapshots]
 
-By default, a restore request attempts to restore all indices and data
-streams in the snapshot, including system indices. If your cluster already
-contains one or more of these system indices, the request will return an error.
-
-To avoid this error, specify the indices and data streams to restore. To exclude
-system indices and other index names that begin with a dot (`.`), append the
-`-.*` wildcard pattern. To restore all indices and data streams except dot
-indices, use `*,-.*`.
+In the restore request, explicitly specify any indices and data streams to
+restore.
 
 [source,console]
 ----
-POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore 
+POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 {
   "indices": "my-index,logs-my_app-default"
 }
@@ -149,7 +150,7 @@ any restored index or data stream.
 
 [source,console]
 ----
-POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore 
+POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 {
   "indices": "my-index,logs-my_app-default",
   "rename_pattern": "(.+)",
@@ -205,17 +206,94 @@ POST _reindex
 ----
 // TEST[continued]
 
+[discrete]
+[[restore-exclude-system-indices]]
+==== Exclude system indices
+
+Some {es} features, such as the <<geoip-processor,GeoIP processor>>,
+automatically create system indices at startup. To avoid naming conflicts with
+these indices, use the `-.*` wildcard pattern to exclude system indices and
+other dot (`.`) indices from your restore request.
+
+For example, the following request uses the `*,-.*` wildcard pattern to restore
+all indices and data streams except dot indices.
+
+[source,console]
+----
+POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
+{
+  "indices": "*,-.*"
+}
+----
+// TEST[setup:setup-snapshots]
+// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
+// TEST[s/_restore/_restore?wait_for_completion=true/]
+
+[discrete]
+[[restore-feature-state]]
+=== Restore a feature state
+
+You can restore a feature state to recover system indices, system data streams,
+and other configuration data for a feature from a snapshot. Restoring a feature
+state is the preferred way to restore system indices and system data streams.
+
+If you restore a snapshot's cluster state, the operation restores all feature
+states in the snapshot by default. Similarly, if you don't restore a snapshot's
+cluster state, the operation doesn't restore any feature states by default. You
+can also choose to restore only specific feature states from a snapshot,
+regardless of the cluster state.
+
+To view a snapshot's feature states, use the get snapshot API.
+
+[source,console]
+----
+GET _snapshot/my_repository/my_snapshot_2099.05.06
+----
+// TEST[setup:setup-snapshots]
+
+The response's `feature_states` property contains a list of features in the
+snapshot as well as each feature's indices.
+
+To restore a specific feature state from the snapshot, specify the
+`feature_name` from the response in the restore snapshot API's
+<<restore-snapshot-api-feature-states,`feature_states`>> parameter. When you
+restore a feature state, {es} closes and overwrites the feature's existing
+indices.
+
+WARNING: Restoring the `security` feature state overwrites system indices
+used for authentication. If you use {ess}, ensure you have access to the {ess}
+Console before restoring the `security` feature state. If you run {es} on your
+own hardware, <<restore-create-file-realm-user,create a superuser in the file
+realm>> to ensure you'll still be able to access your cluster.
+
+To avoid accidentally restoring system indices, system data streams, and other
+dot indices, append the `-.*` wildcard pattern to the `indices` value.
+
+[source,console]
+----
+POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
+{
+  "indices": "*,-.*",
+  "feature_states": [ "geoip" ]
+}
+----
+// TEST[setup:setup-snapshots]
+// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
+// TEST[s/_restore/_restore?wait_for_completion=true/]
+// TEST[s/",/"/]
+// TEST[s/"feature_states": \[ "geoip" \]//]
+
 [discrete]
 [[restore-entire-cluster]]
 === Restore an entire cluster
 
 In some cases, you need to restore an entire cluster from a snapshot, including
-the cluster state and all system indices. These cases should be rare, such as in
+the cluster state and all feature states. These cases should be rare, such as in
 the event of a catastrophic failure.
 
-Restoring an entire cluster involves deleting important system indices, including
-those used for authentication. Consider whether you can restore specific indices
-or data streams instead.
+Restoring an entire cluster involves deleting important system indices,
+including those used for authentication. Consider whether you can restore
+specific indices or data streams instead.
 
 If you're restoring to a different cluster, see <<restore-different-cluster>>
 before you start.
@@ -235,6 +313,18 @@ the cluster.
 . Temporarily stop indexing and turn off the following features:
 +
 --
+* GeoIP database downloader
++
+[source,console]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "ingest.geoip.downloader.enabled": false
+  }
+}
+----
+
 * ILM
 +
 [source,console]
@@ -265,19 +355,18 @@ POST _ml/set_upgrade_mode?enabled=false
 // TEST[continued]
 ////
 
-////
+* Monitoring
++
 [source,console]
 ----
 PUT _cluster/settings
 {
   "persistent": {
-    "xpack.monitoring.collection.enabled": true
+    "xpack.monitoring.collection.enabled": false
   }
 }
 ----
 // TEST[warning:[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.]
-// TEST[continued]
-////
 
 * Watcher
 +
@@ -295,23 +384,29 @@ POST _watcher/_start
 ////
 --
 
-. If you use {es} security features, log in to a node host, navigate to the {es}
+
+. {blank}
++
+--
+[[restore-create-file-realm-user]]
+If you use {es} security features, log in to a node host, navigate to the {es}
 installation directory, and add a user with the `superuser` role to the file
 realm using the <<users-command,`elasticsearch-users`>> tool.
-+
+
 For example, the following command creates a user named `restore_user`.
-+
+
 [source,sh]
 ----
 ./bin/elasticsearch-users useradd restore_user -p my_password -r superuser
 ----
-+
+
 Use this file realm user to authenticate requests until the restore operation is
 complete.
+--
 
 . Use the <<cluster-update-settings,cluster update settings API>> to set
 <<action-destructive-requires-name,`action.destructive_requires_name`>> to
-`false`. This lets you delete indices and data streams using wildcards.
+`false`. This lets you delete data streams and indices using wildcards.
 +
 [source,console]
 ----
@@ -324,24 +419,28 @@ PUT _cluster/settings
 ----
 // TEST[setup:setup-snapshots]
 
-. Delete existing indices and data streams on the cluster.
+. Delete all existing data streams on the cluster.
 +
 [source,console]
 ----
-# Delete all indices
-DELETE *
+DELETE _data_stream/*?expand_wildcards=all
+----
+// TEST[continued]
 
-# Delete all data streams
-DELETE _data_stream/*
+. Delete all existing indices on the cluster.
++
+[source,console]
+----
+DELETE *?expand_wildcards=all
 ----
 // TEST[continued]
 
-. Restore the entire snapshot, including the cluster state. This also restores
-any system indices in the snapshot.
+. Restore the entire snapshot, including the cluster state. By default,
+restoring the cluster state also restores any feature states in the snapshot.
 +
 [source,console]
 ----
-POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore 
+POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 {
   "indices": "*",
   "include_global_state": true
@@ -354,6 +453,19 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 features you stopped:
 +
 --
+* GeoIP database downloader
++
+[source,console]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "ingest.geoip.downloader.enabled": true
+  }
+}
+----
+//TEST[s/true/false/]
+
 * ILM
 +
 [source,console]
@@ -368,6 +480,20 @@ POST _ilm/start
 POST _ml/set_upgrade_mode?enabled=false
 ----
 
+* Monitoring
++
+[source,console]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "xpack.monitoring.collection.enabled": true
+  }
+}
+----
+// TEST[warning:[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.]
+// TEST[s/true/false/]
+
 * Watcher
 +
 [source,console]