KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker #98115

toughcoding · 2023-08-01T19:26:50Z

Elasticsearch Version

8.9.0

Installed Plugins

No response

Java Version

bundled

OS Version

ubuntu:20.04 Linux 753c3b2604ff 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Problem Description

When starting Elasticsearch as a docker container with KEYSTORE_PASSWORD and ELASTIC_PASSWORD set and keystore being password secured, it does not working properly as bootstrap.password is not setup correctly.
When checking value of bootstrap.password it is returning empty string. Thus command line 54
(echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')

is not causing bootstrap.password entry being created although running these commands manually in the container are successfull.

In Contrary Running with non-encrypted keystore making bootstrap.password to be setup properly like below

docker run --rm \
-d \
-e ELASTIC_PASSWORD="123456" \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Steps to Reproduce

Create container only to make encrypted keystore

docker run --rm \
--name elk \
-d \
-v esconf:/usr/share/elasticsearch/config \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Run command to setup password for already created keystore setting password as 123456
docker exec -it elk elasticsearch-keystore passwd

docker exec -it elk elasticsearch-keystore list  
Enter password for the elasticsearch keystore : 
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

Stop container
docker stop elk

Start another container with password variables

docker run --rm \
--name elk \
-d \
-v esconf:/usr/share/elasticsearch/config \
-e ELASTIC_PASSWORD="987654" \
-e KEYSTORE_PASSWORD="123456" \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Checking if bootstrap password entry got created

docker exec -it elk elasticsearch-keystore list
Enter password for the elasticsearch keystore : 
bootstrap.password
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

Run command to display bootstrap.password that normally should be equal to ELASTIC_PASSWORD
docker exec -it elk elasticsearch-keystore show bootstrap.password

BUT it is returning empty string thus bug.

Logs (if relevant)

Beginning of logs after starting second container. Script asking twice for keystore password.
docker run --rm
--name elk
-v esconf:/usr/share/elasticsearch/config
-e ELASTIC_PASSWORD="123456"
-e KEYSTORE_PASSWORD="123456"
docker.elastic.co/elasticsearch/elasticsearch:8.9.0
Enter password for the elasticsearch keystore : Enter password for the elasticsearch keystore : {"@timestamp":"2023-08-01T19:08:33.302Z", "log.level": "INFO",

The text was updated successfully, but these errors were encountered:

toughcoding · 2023-08-01T19:30:34Z

There was related discussion for it on forum.
https://discuss.elastic.co/t/keystore-password-file/339627/4

elasticsearchmachine · 2023-08-02T14:46:35Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine · 2023-08-02T14:47:53Z

Pinging @elastic/es-delivery (Team:Delivery)

mark-vieira · 2023-08-22T21:19:32Z

@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?

Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
	at java.base/java.nio.file.Files.move(Files.java:1429)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
	at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
	Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option
		at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500)
		at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861)
		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
		at java.base/java.nio.file.Files.move(Files.java:1429)
		at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)

ywangd · 2023-08-24T07:48:14Z

I think this is related to #76124

The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with

COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")"
echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo'

# The following command shows empty for `foo`
./bin/elasticsearch-keystore show foo

Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content.
I think it needs attention from @elastic/es-core-infra
Also ping @albertzaharovits since he was the author of the linked PR.

toughcoding · 2023-08-27T15:16:09Z

@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?

Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
	at java.base/java.nio.file.Files.move(Files.java:1429)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
	at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
	Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option
		at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500)
		at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861)
		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
		at java.base/java.nio.file.Files.move(Files.java:1429)
		at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)

To make it work like you wish I am running first container with mounted volume into config and generate password protected keystore. Then secondly I am stopping that container and starting new container with mounted config catalog not just keystore file. So I am using volumes therefore not bind mounts. Elastic team do not recommend to bind mount keystore file mentioning it can cause issues.

toughcoding · 2023-08-27T15:17:59Z

I think this is related to #76124

The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with
COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")"
echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo'

# The following command shows empty for `foo`
./bin/elasticsearch-keystore show foo
Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content. I think it needs attention from @elastic/es-core-infra Also ping @albertzaharovits since he was the author of the linked PR.

Yes it is not due to docker itself but on high level running docker does not work as you might expect from documentation.

rjernst · 2023-08-28T20:18:34Z

I was positive we had keystore tests for this exact scenario. However, upon closer inspection I found that the tests are bogus for a variety of reasons. Unfortunately there are a couple places where we use buffered readers which break this. Fixing it is non trivial; the internal Terminal abstraction needs more work to cleanup how we handle raw input.

elasticsearchmachine · 2023-08-28T22:32:16Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

da1910 · 2023-12-04T15:46:24Z

Its been a little while but I think I'm seeing this same issue when trying to use a precreated keystore, we're providing the password to the keystore in the KEYSTORE_PASSWORD environment variable, but the entrypoint continues to prompt for a password.

If I exec into a running container I can see the variable is set, and if I then run /bin/elasticsearch and enter the keystore password things proceed. But it's not respecting the environment variable as far as I can see. Is there a workaround in the mean time?

toughcoding · 2023-12-12T23:21:50Z

@da1910 You can check my article about this. Maybe you will find some inspiration from commands. Otherwise let's wait for Elastic to come back with suggestions.

When reading a string value from stdin the keystore add command currently looks directly at stdin. However, stdin may also be consumed while reading the keystore password. This commit changes the add command to use the reader from the termainl instead of looking at stdin directly. closes elastic#98115

When reading a string value from stdin the keystore add command currently looks directly at stdin. However, stdin may also be consumed while reading the keystore password. This commit changes the add command to use the reader from the termainl instead of looking at stdin directly. closes #98115

When reading a string value from stdin the keystore add command currently looks directly at stdin. However, stdin may also be consumed while reading the keystore password. This commit changes the add command to use the reader from the termainl instead of looking at stdin directly. closes #98115 Co-authored-by: Elastic Machine <[email protected]>

toughcoding added >bug needs:triage Requires assignment of a team area label labels Aug 1, 2023

astefan added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Aug 2, 2023

elasticsearchmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. and removed needs:triage Requires assignment of a team area label labels Aug 2, 2023

elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Aug 2, 2023

elasticsearchmachine removed the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Aug 2, 2023

mark-vieira added the :Core/Infra/CLI CLI utilities, scripts, and infrastructure label Aug 28, 2023

elasticsearchmachine added the Team:Core/Infra Meta label for core/infra team label Aug 28, 2023

williamrandolph assigned williamrandolph and unassigned williamrandolph Dec 6, 2023

rjernst mentioned this issue Jan 8, 2024

Ensure test clusters use keystore cli tool flag for reading from stdin #104102

Closed

h3xcat mentioned this issue Nov 20, 2024

Fix ELASTIC_PASSWORD with KEYSTORE_PASSWORD docker usage #117185

Closed

rjernst mentioned this issue Apr 11, 2025

Use terminal reader in keystore add command #126729

Merged

rjernst closed this as completed in #126729 Apr 14, 2025

rjernst closed this as completed in b47bd3a Apr 14, 2025

rjernst mentioned this issue Apr 16, 2025

Use terminal reader in keystore add command (#126729) #126963

Merged

rjernst mentioned this issue Apr 16, 2025

Use terminal reader in keystore add command (#126729) #126964

Merged

rjernst mentioned this issue Apr 16, 2025

Use terminal reader in keystore add command (#126729) #126965

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker #98115

KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker #98115

toughcoding commented Aug 1, 2023

toughcoding commented Aug 1, 2023

elasticsearchmachine commented Aug 2, 2023

elasticsearchmachine commented Aug 2, 2023

mark-vieira commented Aug 22, 2023

ywangd commented Aug 24, 2023

toughcoding commented Aug 27, 2023

toughcoding commented Aug 27, 2023

rjernst commented Aug 28, 2023

elasticsearchmachine commented Aug 28, 2023

da1910 commented Dec 4, 2023

toughcoding commented Dec 12, 2023

KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker #98115

KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker #98115

Comments

toughcoding commented Aug 1, 2023

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

toughcoding commented Aug 1, 2023

elasticsearchmachine commented Aug 2, 2023

elasticsearchmachine commented Aug 2, 2023

mark-vieira commented Aug 22, 2023

ywangd commented Aug 24, 2023

toughcoding commented Aug 27, 2023

toughcoding commented Aug 27, 2023

rjernst commented Aug 28, 2023

elasticsearchmachine commented Aug 28, 2023

da1910 commented Dec 4, 2023

toughcoding commented Dec 12, 2023