Orphaned CloudStack VM's present in slow CloudStack environments

[maxdrib]

/kind bug

What steps did you take and what happened:
In slow CloudStack environments, we have observed the following race condition:

1. User requests scale up of resources
2. CAPI machine is created
3. CloudStackMachine is created
4. CAPC begins creating a VM
5. Before VM creation finishes, MachineHealthCheck triggers and CAPI machine is removed
6. Since VM creation did not complete in the CAPC machine controller, Spec.InstanceID is not set on the CloudStackMachine resource, and hence when the CloudStackMachine is being ReconcileDeleted, the call to DestroyVMInstance is skipped. This results in a Cloudstack VM and K8s node existing on the cluster without the corresponding CAPI/CAPC Machine resources to manage it.

What did you expect to happen:
I expected there not to be any rogue VM's and a 1:1 mapping between CloudStackMachine and CloudStack VM's.

Anything else you would like to add:
Resolution is to modify the VM deletion logic. in cloudstackmachine_controller, if Spec.InstanceID is not set, look it up from VM name by calling listVirtualMachines. Then proceed to call DestroyVM in all situations.

Environment:

• Cluster-api-provider-cloudstack version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[maxdrib]

This issue is also similar to kubernetes-sigs/cluster-api#7237 in that it's due to MHC acting aggressively during VM creation, and causing a race condition between the CAPI resources and the K8s/ACS resources. With #190 being fixed, we may still see orphaned K8s nodes, but they will not have associated ACS VM's associated with them. Addressing aws/eks-anywhere#3918 may also help minimize the number of orphaned K8s nodes.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.