CAPi doesnt wait for CSI volume unbinding

[MaxRink]

What steps did you take and what happened:
Ive rolled my workers which had volumes provisioned by the vsphere CSI attached.
Some of those volumes did not detach properly as CAPI was too fast and removed the nodes before the csi controller could fully detach the volumes.

What did you expect to happen:
CAPI waits until volumes are detached.

Anything else you would like to add:
We had a small discussion in the CAPV slack ( https://kubernetes.slack.com/archives/CKFGK3SSD/p1622198292045000 )
@jzhoucliqr has already an patch ( spectrocloud@c340e68 )

Environment:

• Cluster-api version: alpha3

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[enxebre]

@MaxRink what does signal the csi controller to detach the volumes? Should the vSphereMachine controller ensure detach happens gracefully orthogonally to the csi controller?

[MaxRink]

The pod eviction, after the pods are stopped the csi controller will detach the voumes so they can be rebound on another node.
Right now, as far as capi is concerned, after all pods are terminated the node can be safely deleted, getting in the way of the csi nod deamonset and the controller coordinating the volume detachment

[fabriziopandini]

I'm not sure this is really a CAPI bug, given that CAPI is not aware of the type of storage providers in use in each cluster.
Nevertheless, if I got the problem right, machine lifecycle hooks could be a valid solution here; they provide an extension point each provider/each User can exploit to check additional conditions before allowing node deletion, like in this case CSI volumes cleanup.
cc @yastij @gab-satchi

[yastij]

generally speaking I think CAPI should ensure that volumes are properly detached before deleting the machines. That can be done regardless of the storage provider e.g. @jzhoucliqr.

@randomvariable - IIRC CAPA do not have this problem as non-root volumes are detached and preserved on instance termination right ?

[MaxRink]

Its not only the vsphere CSI which might have that issue btw.
Other providers like NetApp Trident, Pure PSO and other iSCSI based provisionbers also rely on properly detaching before node deletion.

[vincepri]

@yastij Are you thinking about inspecting a node and relative Volumes' status?

[vincepri]

/lifecycle awaiting-more-evidence
/milestone Next

[vincepri]

/milestone v0.4