A node that's not found triggers rate limited infinite loop

[jhorwit2]

While deploying a new cluster I mismatched my hostname to the display name which caused the CCM to get into a rate limiting infinite loop since the node controllers backoff retry is fairly fast compared to the rate limits on OCI's API.

We need to start rate limiting requests in the client which is similar to what Azure does. We should also implement additional delay on 429 responses to help the CCM recover.

[prydie]

See Azure cloud provider in core: https://github.com/kubernetes/kubernetes/blob/4b26d70dafeb6ed046fc07e6e223600fc7e55e7e/pkg/cloudprovider/providers/azure/azure.go#L190

[simonlord]

I feel we want the backoff/retry functionality for this to be more useful, otherwise we'll just see lots of RateLimitErrors in the logs? Whereas what we want is for progress to be made even if it's slow (e.g., http 429: https://docs.us-phoenix-1.oraclecloud.com/Content/API/References/apierrors.htm) . Do we also want it on a resource by resource basis - e.g., compute and network clients should have their own ratelimiter as they're seperate server side apis (this is a guess though)?