Narrow period of Shrink action in which ILM prevents stopping

[dakrone]

Prior to this change, we would prevent stopping of ILM if the index was
anywhere in the shrink action. This commit changes
IndexLifecycleService to allow stopping when in any of the innocuous
steps during shrink. This changes ILM only to prevent stopping if
absolutely necessary.

Resolves #43253

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

With these changes, it becomes possible for ILM to stop while shards are allocated to a specific node _id, which could cause non-obvious problems if some new allocation rules are added to relocate those shards off of that node because (for example) that node is going to be replaced with a new one.

There's not an easy solution here, because if we expand the list of steps that will block ILM from stopping, it becomes possible for ILM to refuse to stop because the node it picked to allocate all the shards to doesn't have enough disk space to hold all the shards.

I don't think that should block this PR, but I wanted to bring up the possibility.

[dakrone]

Thanks @gwbrown, I think I addressed all of your feedback

With these changes, it becomes possible for ILM to stop while shards are allocated to a specific node _id, which could cause non-obvious problems if some new allocation rules are added to relocate those shards off of that node because (for example) that node is going to be replaced with a new one.

This is only for automated automatic allocation rules though, not really something we can handle from within ILM. It's also possible that ILM could fail to ever finish the _id relocation due to other pre-existing rules, in which case it'd never be able to stop.

There's not an easy solution here, because if we expand the list of steps that will block ILM from stopping, it becomes possible for ILM to refuse to stop because the node it picked to allocate all the shards to doesn't have enough disk space to hold all the shards.

Yeah, I agree, however, I think I'd favor being able to stop over never stopping at all due to some other condition preventing ILM from moving forward in the step.

I don't think that should block this PR, but I wanted to bring up the possibility.

I agree we should discuss the logic with folks who want to do automated relocation dances (ie: Cloud, ECE, k8s) to see what they should do in those cases, but for now I think this makes the situation much easier to reason about and less likely to be stuck waiting forever for shutdown.

[gwbrown]

LGTM, thanks @dakrone!