MachineSet Inplace update does not work during machine deletion

[davidvossel]

What steps did you take and what happened?

There are values on a machine spec that impact how a machine is deleted. These values can not be modified inplace from the MachineDeployment or MachineSet once a machine is in the process of being deleted. This can result in machines that cannot be torn down successfully.

Here's an example scenario.

1. Make a MachineDeployment with the default nodeDrainTimeout of 0s (which blocks indefinitely if a machine's node can't be drained)
2. Scale down the MachineDeployment in a way that causes a PDB to block a machine's node drain
3. Machine will get stuck in deletion forever

The scenario above is expected and not a bug. However if I add a 4th step here to try and unblock the machine teardown by setting nodeDrainTimeout to something like 5m, the Machine will still be wedged forever despite nodeDrainTimeout being an inplace update field now #8111

This means once a machine is being torn down, trying to modify any fields that influence how the machine is torn down (like nodeDrainTimeout) will not be applied in place.

What did you expect to happen?

modifying nodeDrainTimeout on a machineDeployment should update the MachineSet and all machines in place... This inplace update doesn't occur when the machines are being deleted though.

Cluster API version

main branch

Kubernetes version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[enxebre]

/triage accepted

[fabriziopandini]

/priority important-longterm