1.3.11 IAM-Controller Instance Profile CRD Bug

[cam634]

Describe the bug
When updating the IAM Controller we noticed excess of logging around a CRD for instance profile we do not use the CRD for instance profile however it is producing noisy logs. The controller still works however just creates excess noise. This seen when upgrading to the ACK IAM 1.3.11.

After we went to version 1.3.1 and no longer see this issue

{"level":"error","ts":"2024-08-30T13:56:53.959Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"InstanceProfile.iam.services.k8s.aws","error":"no matches for kind \"InstanceProfile\" in version \"iam.services.k8s.aws/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:71\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:87\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:88\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:64"}
{"level":"error","ts":"2024-08-30T14:16:28.357Z","msg":"Could not wait for Cache to sync","controller":"instanceprofile","controllerGroup":"iam.services.k8s.aws","controllerKind":"InstanceProfile","error":"failed to wait for instanceprofile caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.InstanceProfile","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:203\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:229\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:226"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"error","ts":"2024-08-30T14:16:28.357Z","msg":"Could not wait for Cache to sync","controller":"instanceprofile","controllerGroup":"iam.services.k8s.aws","controllerKind":"InstanceProfile","error":"failed to wait for instanceprofile caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.InstanceProfile","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:203\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:229\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:226"}
{"level":"error","ts":"2024-08-30T14:16:28.357Z","msg":"error received after stop sequence was engaged","error":"failed to wait for instanceprofile caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.InstanceProfile","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:499"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"openidconnectprovider","controllerGroup":"iam.services.k8s.aws","controllerKind":"OpenIDConnectProvider"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"openidconnectprovider","controllerGroup":"iam.services.k8s.aws","controllerKind":"OpenIDConnectProvider"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"policy","controllerGroup":"iam.services.k8s.aws","controllerKind":"Policy"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"role","controllerGroup":"iam.services.k8s.aws","controllerKind":"Role"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"group","controllerGroup":"iam.services.k8s.aws","controllerKind":"Group"}
{"level":"info","ts":"2024-08-30T14:16:28.357Z","msg":"All workers finished","controller":"group","controllerGroup":"iam.services.k8s.aws","controllerKind":"Group"}

Steps to reproduce
Install Helm chart

view pod logs

Expected outcome
Only log when there are actual errors or syncs

Environment

• Kubernetes version 1.29
• Using EKS (yes/no), if so version? Yes, 1.29
• AWS service targeted (S3, RDS, etc.) IAM

[cam634]

I am having the same issue in my dev env on 1.3.1 it worked on one dev going from 1.3.11 down to 1.3.1 it installed the CRD correctly in env then broke on another
the instanceprofiles.iam.services.k8s.aws is not getting installed for some reason

[gecube]

@cam634 Hi! I just realised that I am already running 1.3.12... Did you try it?

Also did you get the issue in fresh install or only when updating the existing one? If latter, what method do you use for the installation of ack?

I faced some issues with upgrading charts and described them in #2007

[cam634]

@gecube Our issue is related with the helm terraform provider sadly hashicorp/terraform-provider-helm#944 It does not update the CRD and install the new crd.

[gecube]

@cam634 it is sad to hear that there is an issue in tf provider. I hope it would be solved somehow. Maybe it would be useful to you, but I don't recommend using TF to manage k8s cluster. Instead of it we create the k8s cluster by TF, instantly bootstrap some gitops tooling of your choice (I use FluxCD, but ArgoCD will work for you as well). And it solves the issue as both FluxCD and ArgoCD don't have issues with CRD updates. Maybe it would be helpful for you and give some ideas how to make a workaround.