Adjust docs for password protected keystore

docs/reference/cluster.asciidoc

@@ -109,3 +109,5 @@ include::cluster/nodes-hot-threads.asciidoc[]
 include::cluster/allocation-explain.asciidoc[]
 
 include::cluster/voting-exclusions.asciidoc[]
+
+include::cluster/nodes-reload-secure-settings.asciidoc[]

docs/reference/cluster/nodes-reload-secure-settings.asciidoc

@@ -1,13 +1,7 @@
 [[cluster-nodes-reload-secure-settings]]
 == Nodes Reload Secure Settings
 
-The cluster nodes reload secure settings API is used to re-read the
-local node's encrypted keystore. Specifically, it will prompt the keystore
-decryption and reading across the cluster. The keystore's plain content is
-used to reinitialize all compatible plugins. A compatible plugin can be
-reinitialized without restarting the node. The operation is
-complete when all compatible plugins have finished reinitializing. Subsequently,
-the keystore is closed and any changes to it will not be reflected on the node.
+The cluster nodes reload secure settings API is used to re-load the keystore on each node.
 
 [source,js]
 --------------------------------------------------
@@ -22,9 +16,41 @@ The first command reloads the keystore on each node. The seconds allows
 to selectively target `nodeId1` and `nodeId2`. The node selection options are
 detailed <<cluster-nodes,here>>.
 
-Note: It is an error if secure settings are inconsistent across the cluster
-nodes, yet this consistency is not enforced whatsoever. Hence, reloading specific
-nodes is not standard. It is only justifiable when retrying failed reload operations.
+NOTE: It is an error if secure settings are inconsistent across the cluster nodes, yet this consistency is not enforced
+whatsoever. Hence, reloading specific nodes is not standard.It is only justifiable when retrying failed reload operations.
+
+==== Reload Password Protected Secure Settings
+
+When the elasticsearch keystore is password protected and not simply obfuscated, the password for the keystore needs
+to be provided in the request to reload the secure settings.
+Reloading the settings for the whole cluster assumes that all nodes' keystores are protected with the same password
+and is only allowed when {ref}/configuring-tls.html#tls-transport[node to node communications are encrypted]
+
+[source,js]
+--------------------------------------------------
+POST _nodes/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The common password that the elasticsearch keystore is encrypted with in every node of the cluster.
+
+Alternatively the secure settings can be reloaded on a per node basis, locally accessing the API and passing the
+node-specific elasticsearch keystore password.
+
+[source,js]
+--------------------------------------------------
+POST _nodes/_local/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The password that the elasticsearch keystore is encrypted with on the local node.
+
 
 [float]
 [[rest-reload-secure-settings]]

docs/reference/setup/secure-settings.asciidoc

@@ -10,16 +10,12 @@ NOTE: All commands here should be run as the user which will run Elasticsearch.
 IMPORTANT: Only some settings are designed to be read from the keystore. However,
 the keystore has no validation to block unsupported settings.
 Adding unsupported settings to the keystore will cause {es}
-Additional unsupported settings being added to the keystore will cause Elasticsearch
 to fail to start. See documentation for each setting to see if it is supported
 as part of the keystore.
 
 NOTE: All the modifications to the keystore take affect only after restarting
 Elasticsearch.
 
-NOTE: The elasticsearch keystore currently only provides obfuscation. In the future,
-password protection will be added.
-
 These settings, just like the regular ones in the `elasticsearch.yml` config file,
 need to be specified on each node in the cluster. Currently, all secure settings
 are node-specific settings that must have the same value on every node.
@@ -32,16 +28,33 @@ To create the `elasticsearch.keystore`, use the `create` command:
 
 [source,sh]
 ----------------------------------------------------------------
-bin/elasticsearch-keystore create
+bin/elasticsearch-keystore create -p
+----------------------------------------------------------------
+
+You will be prompted to enter the keystore password and the file `elasticsearch.keystore` will be created alongside `elasticsearch.yml`, protected with the password you specified.
+
+NOTE: If you don't specify the `-p` flag or if you enter an empty password, the elasticsearch keystore will be obfuscated but not password protected.
+
+[float]
+[[changing-keystore-password]]
+=== Changing the password of the keystore
+
+To change the password of the `elasticsearch.keystore`, use the `passwd` command.
+If the elasticsearch keystore is password protected, you will be prompted to enter the current password and then enter the new one
+
+[source,sh]
+----------------------------------------------------------------
+bin/elasticsearch-keystore passwd
 ----------------------------------------------------------------
 
-The file `elasticsearch.keystore` will be created alongside `elasticsearch.yml`.
+You can use the `passwd` subcommand to set a password to a previously obfuscated only keystore, but also to remove the password from an encrypted keystore by setting it to an empty string.
 
 [float]
 [[list-settings]]
 === Listing settings in the keystore
 
-A list of the settings in the keystore is available with the `list` command:
+A list of the settings in the keystore is available with the `list` command.
+If the elasticsearch keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -52,8 +65,8 @@ bin/elasticsearch-keystore list
 [[add-string-to-keystore]]
 === Adding string settings
 
-Sensitive string settings, like authentication credentials for cloud
-plugins, can be added using the `add` command:
+Sensitive string settings, like authentication credentials for cloud plugins, can be added using the `add` command.
+If the elasticsearch keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -68,12 +81,22 @@ through stdin, use the `--stdin` flag:
 cat /file/containing/setting/value | bin/elasticsearch-keystore add --stdin the.setting.name.to.set
 ----------------------------------------------------------------
 
+You can overwrite existing entries in the keystore by setting the `-f` flag
+
+[source,sh]
+----------------------------------------------------------------
+bin/elasticsearch-keystore add -f the.existing.setting.name.to.set
+----------------------------------------------------------------
+
+NOTE: The `-f` flag will also force the creation of an obfuscated-only keystore, if one doesn't already exist.
+
 [float]
 [[add-file-to-keystore]]
 === Adding file settings
 You can add sensitive files, like authentication key files for cloud plugins,
 using the `add-file` command. Be sure to include your file path as an argument
 after the setting name.
+If the elasticsearch keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -84,7 +107,8 @@ bin/elasticsearch-keystore add-file the.setting.name.to.set /path/example-file.j
 [[remove-settings]]
 === Removing settings
 
-To remove a setting from the keystore, use the `remove` command:
+To remove a setting from the keystore, use the `remove` command.
+If the elasticsearch keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -98,24 +122,29 @@ bin/elasticsearch-keystore remove the.setting.name.to.remove
 Just like the settings values in `elasticsearch.yml`, changes to the
 keystore contents are not automatically applied to the running
 elasticsearch node. Re-reading settings requires a node restart.
-However, certain secure settings are marked as *reloadable*. Such settings
-can be re-read and applied on a running node.
+However, certain secure settings are marked as *reloadable*.
+Such settings can be {ref}/nodes-reload-secure-settings.html[re-read and applied on a running node].
 
 The values of all secure settings, *reloadable* or not, must be identical
 across all cluster nodes. After making the desired secure settings changes,
 using the `bin/elasticsearch-keystore add` command, call:
 [source,js]
 ----
 POST _nodes/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
 ----
-// CONSOLE
-This API will decrypt and re-read the entire keystore, on every cluster node,
-but only the *reloadable* secure settings will be applied. Changes to other
-settings will not go into effect until the next restart. Once the call returns,
-the reload has been completed, meaning that all internal datastructures dependent
+// NOTCONSOLE
+
+<1> The password that the elasticsearch keystore is encrypted with.
+
+This API will decrypt and re-read the entire keystore, on every cluster node, but only the *reloadable* secure settings will be applied.
+Changes to other settings will not go into effect until the next restart.
+Once the call returns, the reload has been completed, meaning that all internal data structures dependent
 on these settings have been changed. Everything should look as if the settings
 had the new value from the start.
 
-When changing multiple *reloadable* secure settings, modify all of them, on
-each cluster node, and then issue a `reload_secure_settings` call, instead
+When changing multiple *reloadable* secure settings, modify all of them, on each cluster node, and then issue a
+{ref}/nodes-reload-secure-settings.html[`reload_secure_settings`] call, instead
 of reloading after each modification.