Cluster deletion fails when a cluster has a LoadBalancer Service type

[dkoshkin]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

1. Create an AWS cluster following https://cluster-api.sigs.k8s.io/user/quick-start.html
2. Create a Service of type LoadBalancer
3. Delete the cluster

The cluster does not get fully deleted.

I see that the ELB for the Service does get deleted:

I0517 15:25:09.694986       1 loadbalancer.go:588] controller/awscluster "msg"="Deleted AWS cloud provider load balancers" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
I0517 15:25:09.718863       1 loadbalancer.go:185] controller/awscluster "msg"="Deleted control plane load balancer" "cluster"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" "name"="dkoshkin-elbs-apiserver"
I0517 15:25:10.288928       1 loadbalancer.go:588] controller/awscluster "msg"="Deleted AWS cloud provider load balancers" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"

And the CAPA SGs get deleted, but not ELB's SG

I0517 15:25:12.219861       1 securitygroups.go:311] controller/awscluster "msg"="Deleted security group" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" "kind"="cluster managed" "security-group-id"="sg-039394268c837ac54"
I0517 15:25:14.160000       1 securitygroups.go:311] controller/awscluster "msg"="Deleted security group" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" "kind"="cluster managed" "security-group-id"="sg-0e7a97e0d1721b8e1"
I0517 15:25:14.615167       1 securitygroups.go:311] controller/awscluster "msg"="Deleted security group" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" "kind"="cluster managed" "security-group-id"="sg-0fc93c6b4f7dc9fd0"

After some time

E0517 15:37:14.099976       1 awscluster_controller.go:230] controller/awscluster "msg"="error deleting network" "error"="failed to delete vpc \"vpc-0a70f71fe3c852939\": DependencyViolation: The vpc 'vpc-0a70f71fe3c852939' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 7b9b3128-669a-469b-b2ce-000cf4e65602" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
E0517 15:37:14.221830       1 controller.go:317] controller/awscluster "msg"="Reconciler error" "error"="failed to delete vpc \"vpc-0a70f71fe3c852939\": DependencyViolation: The vpc 'vpc-0a70f71fe3c852939' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 7b9b3128-669a-469b-b2ce-000cf4e65602" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
I0517 15:37:14.223257       1 awscluster_controller.go:198] controller/awscluster "msg"="Reconciling AWSCluster delete" "cluster"="dkoshkin-elbs" "name"="dkoshkin-elbs" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"

What did you expect to happen:
The cluster and cluster's VPC should get deleted.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
I think in addition to deleting the ELBs the controller also needs delete SGs.

We get the kubernetes.io/cluster tag and the name and description, but not much more that indicates thats its an ELB SG

Environment:

• Cluster-api-provider-aws version:
  v1.4.1
• Kubernetes version: (use kubectl version):
  v1.22.8
• OS (e.g. from /etc/os-release):

[k8s-ci-robot]

@dkoshkin: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sedefsavas]

This is being tracked in the cluster-api repo: kubernetes-sigs/cluster-api#3075

This happens when external resources are created by CCM/CSI on the cluster VPC and might be an issue for other providers too, so probably requires a generic/common design.

[dkoshkin]

Thanks @sedefsavas, please feel free to close this one.

[steve-fraser]

We are experiencing this with AWSManagedControlPlane and AwsManagedMachinePool as well

[sedefsavas]

Same issue kubernetes-sigs/cluster-api#3075 applies to EKS clusters as well since we create the VPC for them too (when external infra is not used).

/close

[k8s-ci-robot]

@sedefsavas: Closing this issue.

In response to this:

Same issue kubernetes-sigs/cluster-api#3075 applies to EKS clusters as well since we create the VPC for them too (when external infra is not used).

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.