bin/elasticsearch-reconfigure-node bugs and enhancements

[jkakavas]

This is a meta issue to cover identified bugs and enhancements we want to make to elasticsearch-reconfigure-node. This CLI tool is introduced in 8.0.0 and the main use case it satisfies is to allow users to reconfigure an elasticsearch node that has been installed via a DEB or RPM package and enroll it into an existing cluster.
DEB/RPM installations by default make the assumption that the installed node is the first (or only) one in the cluster and configure TLS for transport and http layers accordingly. This CLI tool allows users to revisit the assumption after installation (before the node is ever started), so that it can enroll to an existing, secured cluster.

• elasticsearch-reconfigure-node should be safer to use. The current behavior is to remove existing configuration and then attempt to enroll to the cluster. We should first attempt to enroll to the cluster and then remove/replace existing configuration, or backup/restore the existing configuration upon failure to enroll, so that we leave the node in a working state in every case.
• elasticsearch-reconfigure-node lacks documentation
• elasticsearch-reconfigure-node is expected to be run as root and wrongly sets file owners for generated keystores, so elasticsearch fails to read them on startup.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)