Multi node security auto-configuration with enrollment mode documentation enhancements

[jkakavas]

Since 8.0.0 and the changes for Security ON by default, when a node starts and gets TLS auto-configuration, we bind the http layer to _site_ and _local_ but transport layer still only to _local_. This is an intentional design decision, and it means that
a) We don't generate an enrollment token for other nodes, as we expect the user needs to perform some additional actions (bind to non localhost address , take care of bootstrap checks ) if/before they can enroll new nodes, so the enrollment token would be probably invalid by the time it would be used.
b) Only nodes on the same host can join the cluster without additional configuration ( set network.host or transport.host to a non localhost address)

Additionally, since we do not bind to non localhost in nodes that enroll to an existing node, enrollment can succeed but the node will fail to join the cluster if the rest of the nodes are in different hosts.

We can

1. Adjust our documentation to make this clear, in the sections where we describe how to manually setup a multi node cluster with our enrollment mode so that users are aware of the limitation and the required actions.
2. (optionally) change our generated configuration to have a commented out #transport.host setting that users can enable, with an associated sentence
3. Adjust the output of our auto-configuration to add information around this
4. Adjust the output of elasticsearch-create-enrollment-token to print a warning when the node that generates the token is only bound to localhost
5. Set transport.host on new nodes that enroll to a cluster if we can determine that they are on different hosts than the existing nodes.
6. allow a --transport.host parameter that woul take an IP Address or hostname and persist that in the configuration of the node that we auto-generate.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[jkakavas]

We discussed this today in our project meeting. My view regarding the potential options is as follows:

• It is still a valid decision to only bind to localhost for the transport layer by default. Single node clusters are a big part of the target user group of security on by default and it doesn't make much sense to me to bind to non localhost and force everyone to deal with bootstrap checks, possibly unnecessarily.

• We will make documentation adjustments to point this out and clarify the behavior and actions required.

• We will adjust elasticsearch-create-enrollment-token tool to print out a warning if the node it is run on only binds to localhost for transport to notify users that nodes from other hosts can't join the cluster. Possibly an instruction to set transport.host too, which accompanied with the commented out section in the configuration, should make it relatively easy to make the change.

• For the first node of the cluster, we will add a commented out section in the node so that it can be more intuitive for the users to figure this out

  # Allow other nodes to join the cluster from localhost and local networks
  # (connections are encrypted and mutually authenticated)
  #transport.host: [_local_, _site_]
  

• For nodes enrolling into an existing cluster, we can decide whether or not it needs to bind to non-localhost addresses by comparing the list of addresses we get for other nodes of the cluster during enrollment and the list of addresses we can get for the host ( NetworkUtils.getAllAddresses() ) we run on. If the list we get from enrollment contains non-localhst addresses that are not in the list of addresses we can figure out for our host, then this means that (some of) the other nodes in the cluster are in different hosts. We can then set transport.host: [_local_, _site_] as otherwise the node won't be able to join the cluster either way.

• I don't think we should add a parameter/flag for this. For enrolling nodes, I believe we can solve this well enough ( see above bullet) .I don't think we should be doing any adjustments/reconfiguration after the node is configured the first time so this flag/parameter would only make sense to be passed the first time. For initial nodes, if you know up front that you want to do a multi node cluster and you know up front that you need to pass that setting when starting elasticsearch for the first time, it's not such a huge benefit compared to editing the config file once and adding it there.

[bytebilly]

Thanks @jkakavas, just a few other points:

If the list we get from enrollment contains non-localhst addresses that are not in the list of addresses we can figure out for our host, then this means that (some of) the other nodes in the cluster are in different hosts. We can then set transport.host: [_local_, _site_] as otherwise the node won't be able to join the cluster either way.

We should consider adding _global_ if the enrollment address is not a site-local one. Restricting to _site_ would not help in that case. Or directly use 0.0.0.0.

In today's world, do we still expect to have public ip addresses directly on the host? The more I think about that, the more I feel that machines always have a private ip, and some sort of gateway handling ip addresses. Using 0.0.0.0 rather than [_local_, _site_] would probably simplify a lot how we can explain it to the customer, where the possible combination of site and global addresses is totally undefiined, as you may have site addresses that are exposed by default to the world (e.g. on Cloud providers). The same applies for the HTTP interface.

What do you think?

I don't think we should add a parameter/flag for this.

I would also prefer this to be automatically configured by the process, since it already modify the configuration file.

[jkakavas]

What do you think?

That's a fair point. If we assume that hosts do not have public routable addresses, the difference between site and global/0.0.0.0 is negligible.
I don't know if we can make that assumption though.. For instance if you spin up an EC2 instance you get a public IP address. If you start ES there, there is a difference between binding to [_site_, _local_] and binding to 0.0.0.0 . The latter makes your elasticsearch node accessible to the world by default and this is most probably not what you want ?
Also for transport layer, I'd assume it's a more common scenario that es nodes don't talk to each other over the public internet, but rather belong in the same network and talk over site local addresses.

[bytebilly]

For instance if you spin up an EC2 instance you get a public IP address.

Uhm, that's not the case. Actually you get a private ip address on the host, and a public ip address on the AWS network edge that masks connections to your private one. The same on GCP, and on many of the Cloud providers. It's also pretty common in home networks and small offices, where there are masquerading rules from the edge router (public ip) to internal hosts (private ip).

And this is exactly the possible confusion that I was describing 🙂
You bind _site_, but you are actually exposed to the world. It's unexpected and potentially a security flaw rather than a protection.

[jkakavas]

Thanks for the correction! I'll take your word for it :)

I don't have a good counter-argument. With the kind of setups you describe, _site_ == _global_ for all intents and purposes so I have no strong objections.

What we should quantify, I think, is whether the case where a host gets a public address (i.e. bare metal, etc ? ) is any common ( or at least common enough to make us consider the difference ). It feels like something we should be running with the rest of the team ( maybe es-core-infra ) so I think you're in a good position to initiate the discussion if that's fine with you

cc @colings86

[bytebilly]

Frankly, last time I saw public IP addresses directly on hosts was at the University. In the good old days, IPv4 were not running out of stock (are they really doing today? 🙂) and the campus had an entire B-class network assigned.

I'm not sure about IPv6 and how Elasticsearch handles it, but I guess that 0.0.0.0 is pretty clear in binding IPv4 only. It is also the default for our Docker containers.

Let's see if anyone has concerns to use it rather than _site_, maybe @qhoxie or @elastic/infosec can provide their perspective on this topic.

[bytebilly]

Bumping the issue to check if there are security concerns in using 0.0.0.0 as the bind host commented value.

[qhoxie]

@bytebilly how would this manifest when you do assign a public IP to an instance? e.g. https://cloud.google.com/compute/docs/ip-addresses#reservedaddress