Password-protected Keystore Feature Branch PR (#51123) (#51510)

* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <[email protected]>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <[email protected]>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <[email protected]>
Co-authored-by: Ioannis Kakavas <[email protected]>

Author: 3 people

Vagrantfile

@@ -479,6 +479,7 @@ JAVA
     ensure curl
     ensure unzip
     ensure rsync
+    ensure expect
 
     installed bats || {
       # Bats lives in a git repository....

distribution/docker/docker-test-entrypoint.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 cd /usr/share/elasticsearch/bin/
-./elasticsearch-users useradd x_pack_rest_user -p x-pack-test-password -r superuser || true 
+./elasticsearch-users useradd x_pack_rest_user -p x-pack-test-password -r superuser || true
 echo "testnode" > /tmp/password
 cat /tmp/password  | ./elasticsearch-keystore add -x -f -v 'xpack.security.transport.ssl.keystore.secure_password'
 cat /tmp/password  | ./elasticsearch-keystore add -x -f -v 'xpack.security.http.ssl.keystore.secure_password'

distribution/docker/src/docker/bin/docker-entrypoint.sh

@@ -57,8 +57,18 @@ if [[ -f bin/elasticsearch-users ]]; then
   # honor the variable if it's present.
   if [[ -n "$ELASTIC_PASSWORD" ]]; then
     [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create)
-    if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
-      (run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
+    if ! (run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent) ; then
+      # keystore is unencrypted
+      if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
+        (run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
+      fi
+    else
+      # keystore requires password
+      if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
+          | elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
+        COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
+        (run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
+      fi
     fi
   fi
 fi
@@ -70,4 +80,4 @@ if [[ "$(id -u)" == "0" ]]; then
   fi
 fi
 
-run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch
+run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch <<<"$KEYSTORE_PASSWORD"

distribution/packages/build.gradle

@@ -231,6 +231,10 @@ Closure commonPackageConfig(String type, boolean oss, boolean jdk) {
       from "${packagingFiles}/systemd/sysctl/elasticsearch.conf"
       fileMode 0644
     }
+    into('/usr/share/elasticsearch/bin') {
+      from "${packagingFiles}/systemd/systemd-entrypoint"
+      fileMode 0755
+    }
 
     // ========= sysV init =========
     configurationFile '/etc/init.d/elasticsearch'

distribution/packages/src/common/scripts/postinst

@@ -108,7 +108,12 @@ if [ "$PACKAGE" = "deb" ]; then
         chmod 660 "${ES_PATH_CONF}"/elasticsearch.keystore
         md5sum "${ES_PATH_CONF}"/elasticsearch.keystore > "${ES_PATH_CONF}"/.elasticsearch.keystore.initial_md5sum
     else
-        /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+        if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
+          echo "### Warning: unable to upgrade encrypted keystore" 1>&2
+          echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
+        else
+          /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+        fi
     fi
 fi
 

distribution/packages/src/common/scripts/posttrans

@@ -11,7 +11,12 @@ if [ ! -f "${ES_PATH_CONF}"/elasticsearch.keystore ]; then
     chmod 660 "${ES_PATH_CONF}"/elasticsearch.keystore
     md5sum "${ES_PATH_CONF}"/elasticsearch.keystore > "${ES_PATH_CONF}"/.elasticsearch.keystore.initial_md5sum
 else
-    /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+    if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
+      echo "### Warning: unable to upgrade encrypted keystore" 1>&2
+      echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
+    else
+      /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+    fi
 fi
 
 ${scripts.footer}

distribution/packages/src/common/systemd/elasticsearch.service

@@ -19,7 +19,7 @@ WorkingDirectory=/usr/share/elasticsearch
 User=elasticsearch
 Group=elasticsearch
 
-ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
+ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet
 
 # StandardOutput is configured to redirect to journalctl since
 # some error messages may be logged in standard output before

distribution/packages/src/common/systemd/systemd-entrypoint

@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# This wrapper script allows SystemD to feed a file containing a passphrase into
+# the main Elasticsearch startup script
+
+if [ -n "$ES_KEYSTORE_PASSPHRASE_FILE" ] ; then
+  exec /usr/share/elasticsearch/bin/elasticsearch "$@" < "$ES_KEYSTORE_PASSPHRASE_FILE"
+else
+  exec /usr/share/elasticsearch/bin/elasticsearch "$@"
+fi

distribution/src/bin/elasticsearch

@@ -20,6 +20,19 @@ if [ -z "$ES_TMPDIR" ]; then
   ES_TMPDIR=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
 fi
 
+# get keystore password before setting java options to avoid
+# conflicting GC configurations for the keystore tools
+unset KEYSTORE_PASSWORD
+KEYSTORE_PASSWORD=
+if ! echo $* | grep -E -q '(^-h |-h$| -h |--help$|--help |^-V |-V$| -V |--version$|--version )' \
+    && "`dirname "$0"`"/elasticsearch-keystore has-passwd --silent
+then
+  if ! read -s -r -p "Elasticsearch keystore password: " KEYSTORE_PASSWORD ; then
+    echo "Failed to read keystore password on console" 1>&2
+    exit 1
+  fi
+fi
+
 ES_JVM_OPTIONS="$ES_PATH_CONF"/jvm.options
 ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_JVM_OPTIONS"`
 
@@ -35,7 +48,7 @@ if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null;
     -Des.bundled_jdk="$ES_BUNDLED_JDK" \
     -cp "$ES_CLASSPATH" \
     org.elasticsearch.bootstrap.Elasticsearch \
-    "$@"
+    "$@" <<<"$KEYSTORE_PASSWORD"
 else
   exec \
     "$JAVA" \
@@ -48,7 +61,7 @@ else
     -cp "$ES_CLASSPATH" \
     org.elasticsearch.bootstrap.Elasticsearch \
     "$@" \
-    <&- &
+    <<<"$KEYSTORE_PASSWORD" &
   retval=$?
   pid=$!
   [ $retval -eq 0 ] || exit $retval

distribution/src/bin/elasticsearch-cli.bat

@@ -25,5 +25,5 @@ set ES_JAVA_OPTS=-Xms4m -Xmx64m -XX:+UseSerialGC %ES_JAVA_OPTS%
   -cp "%ES_CLASSPATH%" ^
   "%ES_MAIN_CLASS%" ^
   %*
-  
+
 exit /b %ERRORLEVEL%

distribution/src/bin/elasticsearch.bat

@@ -4,6 +4,7 @@ setlocal enabledelayedexpansion
 setlocal enableextensions
 
 SET params='%*'
+SET checkpassword=Y
 
 :loop
 FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
@@ -18,6 +19,20 @@ FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
 		SET silent=Y
 	)
 
+	IF "!current!" == "-h" (
+		SET checkpassword=N
+	)
+	IF "!current!" == "--help" (
+		SET checkpassword=N
+	)
+
+	IF "!current!" == "-V" (
+		SET checkpassword=N
+	)
+	IF "!current!" == "--version" (
+		SET checkpassword=N
+	)
+
 	IF "!silent!" == "Y" (
 		SET nopauseonerror=Y
 	) ELSE (
@@ -41,6 +56,18 @@ IF ERRORLEVEL 1 (
 	EXIT /B %ERRORLEVEL%
 )
 
+SET KEYSTORE_PASSWORD=
+IF "%checkpassword%"=="Y" (
+  CALL "%~dp0elasticsearch-keystore.bat" has-passwd --silent
+  IF !ERRORLEVEL! EQU 0 (
+    SET /P KEYSTORE_PASSWORD=Elasticsearch keystore password:
+    IF !ERRORLEVEL! NEQ 0 (
+      ECHO Failed to read keystore password on standard input
+      EXIT /B !ERRORLEVEL!
+    )
+  )
+)
+
 if not defined ES_TMPDIR (
   for /f "tokens=* usebackq" %%a in (`CALL %JAVA% -cp "!ES_CLASSPATH!" "org.elasticsearch.tools.launchers.TempDirectory"`) do set  ES_TMPDIR=%%a
 )
@@ -54,7 +81,20 @@ if "%MAYBE_JVM_OPTIONS_PARSER_FAILED%" == "jvm_options_parser_failed" (
   exit /b 1
 )
 
-%JAVA% %ES_JAVA_OPTS% -Delasticsearch -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" -Des.bundled_jdk="%ES_BUNDLED_JDK%" -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!
+rem windows batch pipe will choke on special characters in strings
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^^=^^^^!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^&=^^^&!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^|=^^^|!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^<=^^^<!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^>=^^^>!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^\=^^^\!
+
+ECHO.!KEYSTORE_PASSWORD!| %JAVA% %ES_JAVA_OPTS% -Delasticsearch ^
+  -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" ^
+  -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" ^
+  -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" ^
+  -Des.bundled_jdk="%ES_BUNDLED_JDK%" ^
+  -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!
 
 endlocal
 endlocal

docs/reference/cluster.asciidoc

@@ -99,6 +99,8 @@ include::cluster/nodes-hot-threads.asciidoc[]
 
 include::cluster/nodes-info.asciidoc[]
 
+include::cluster/nodes-reload-secure-settings.asciidoc[]
+
 include::cluster/nodes-stats.asciidoc[]
 
 include::cluster/pending.asciidoc[]

docs/reference/cluster/nodes-reload-secure-settings.asciidoc

@@ -1,13 +1,11 @@
 [[cluster-nodes-reload-secure-settings]]
-== Nodes Reload Secure Settings
+=== Nodes reload secure settings API
+++++
+<titleabbrev>Nodes reload secure settings</titleabbrev>
+++++
 
-The cluster nodes reload secure settings API is used to re-read the
-local node's encrypted keystore. Specifically, it will prompt the keystore
-decryption and reading across the cluster. The keystore's plain content is
-used to reinitialize all compatible plugins. A compatible plugin can be
-reinitialized without restarting the node. The operation is
-complete when all compatible plugins have finished reinitializing. Subsequently,
-the keystore is closed and any changes to it will not be reflected on the node.
+
+The cluster nodes reload secure settings API is used to re-load the keystore on each node.
 
 [source,console]
 --------------------------------------------------
@@ -21,9 +19,41 @@ The first command reloads the keystore on each node. The seconds allows
 to selectively target `nodeId1` and `nodeId2`. The node selection options are
 detailed <<cluster-nodes,here>>.
 
-Note: It is an error if secure settings are inconsistent across the cluster
-nodes, yet this consistency is not enforced whatsoever. Hence, reloading specific
-nodes is not standard. It is only justifiable when retrying failed reload operations.
+NOTE: {es} requires consistent secure settings across the cluster nodes, but this consistency is not enforced.
+Hence, reloading specific nodes is not standard. It is only justifiable when retrying failed reload operations.
+
+==== Reload Password Protected Secure Settings
+
+When the {es} keystore is password protected and not simply obfuscated, the password for the keystore needs
+to be provided in the request to reload the secure settings.
+Reloading the settings for the whole cluster assumes that all nodes' keystores are protected with the same password
+and is only allowed when {ref}/configuring-tls.html#tls-transport[node to node communications are encrypted]
+
+[source,js]
+--------------------------------------------------
+POST _nodes/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The common password that the {es} keystore is encrypted with in every node of the cluster.
+
+Alternatively the secure settings can be reloaded on a per node basis, locally accessing the API and passing the
+node-specific {es} keystore password.
+
+[source,js]
+--------------------------------------------------
+POST _nodes/_local/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The password that the {es} keystore is encrypted with on the local node.
+
 
 [float]
 [[rest-reload-secure-settings]]