Rolling restart doc uses persistent but should use transient

[msimos]

Elasticsearch version (bin/elasticsearch --version):

6.0.0

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

Step 1 says:

Disable shard allocation.

When you shut down a node, the allocation process waits for one minute before starting to replicate the shards on that node to other nodes in the cluster, causing a lot of wasted I/O. You can avoid racing the clock by disabling allocation before shutting down the node:

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}

However this should be transient settings. Later on it says to use transient to re-enable:

Reenable shard allocation.

Once the node has joined the cluster, reenable shard allocation to start using the node:

PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}

Shouldn't you use transient only? Basically if you were to do a full cluster restart at some point (without disabling allocation), the persistent setting would only apply and allocation is disabled. While transient setting takes precedent over persistent, this is not clearly spelled out in the docs. Also I think it can lead to confusion as to how to enable or disable allocation using transient vs persistent.

[msimos]

Also if you already have the transient settings set to all, then setting persistent in Step 1 I assume has no effect because transient takes precedent.

[voegelas]

We suffered from this documentation mistake too. After a system reboot "cluster.routing.allocation.enable" was set to "none" and newly created indices didn't become available.

[pickypg]

I think the bug here is not that the last setting is set transiently, rather than the full cluster restart and rolling restart documentation are in disagreement.

Rolling restart, step 7 shows:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  }
}

Full cluster restart, step 8 shows:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}

As long as you follow either guide, you should end up with allocation enabled. The only issue here is that a transient setting of "all" is lost following a full cluster restart, but that is a good thing as it's already the first step in a full cluster restart -- so setting it transiently protects you from forgetting to set it to "none" as the first step.

I think the fix should be to change the full cluster restart docs to transiently set the setting in step 8 and then close out this issue.

[astefan]

I agree with this comment, the cluster needs to be brought back to the initial state. Not many users are following the procedure when it comes to a full cluster restart. Usually, they are doing it when they actually upgrade.

In other cases, the cluster is just restarted without any allocation being disabled. When this happens and the cluster comes back no new indices will be allocated and then it is realized that allocation was mysteriously disabled sometime in the past. I am in favor of having

"transient": {
"cluster.routing.allocation.enable": "none"
}

[pickypg]

But when you restart a cluster accidentally, you are in better shape having disabled allocation rather than allowing allocations in an unexpected situation.

It's true that it is not the default, but that's because the default when you startup is to let you get to work. Leaving it set as none leaves you in a safer state whether you intentionally or accidentally restart a node: no allocations happen without you giving permission.

As to transiently setting none, you cannot do this for full cluster restarts because the master disappears and thus allocation happens immediately upon cluster reformation (which is the problem that setting it persistently, and all transiently addresses).

[astefan]

This gh issue was about rolling upgrades and my comment regarding transiently setting it to none applies to rolling upgrades.

I agree it is safer to do it persistently and then reverting it with transiently, but you have the downside of not being aware of the persistent one having been applied at some time and assuming that the cluster should get back to "normal" after restart. Which is not happening until the next day (the logging use case with daily indices) when the user notices that day's indices were not created, Kibana visualizations are not working and all the ingesting flows (LS or not) are accumulating events or, worse, dropping them.

To me, it looks like we are dropping predictability. And safer it may be, but not always (see the logging use case scenario I described above).

[jarro2783]

I think the docs are still wrong for a rolling restart. If I use transient to re-enable, then repeat the procedure for the next node, I am now in the wrong state. If I follow the first step and disable it for persistent, then nothing happens, because it is already disabled, and transient is already enabled. So the docs should either say not to use persistent for a rolling restart at all, or to point out that you need to disable the transient setting for each node.

Then you probably also want it back to the normal state after finishing, meaning that persistent should be set to enabled.

[pickypg]

@jarro2783

That's a great point. Rolling restarts should probably specify none as both persistent and transient (or, for simple matters, just one or the other). At the very least, we must specify in the docs to remove the transient setting between node changes.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

We updated the docs to recommend using persistent throughout the full cluster restart docs in #29670, and in the rolling upgrade docs in #29671, so I think this issue is resolved.