OperatorCondition/Generator controllers churn on missing CSVs

[njhale]

Bug Report

The controllers managing OperatorConditions can churn indefinitely, periodically producing error logs, when a CSV is deleted.

What did you do?

1. kind create cluster
2. make run-local
3. kubectl -n olm edit deployment packageserver, add a finalizer (example: foo.io/bar), save
4. kubectl -n olm delete csv packageserver

What did you expect to see?

A single log entry -- if any -- from the OperatorConditions controllers indicating that the packageserver CSV is missing.

What did you see instead? Under which circumstances?

Periodic log entries on the subject:

1.6462005457872362e+09  ERROR   controllers.operatorcondition   Unable to find operatorcondition        {"request": "olm/packageserver", "error": "OperatorCondition.operators.coreos.com \"packageserver\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
1.646200545787404e+09   ERROR   controller.operatorcondition    Reconciler error        {"reconciler group": "operators.coreos.com", "reconciler kind": "OperatorCondition", "name": "packageserver", "namespace": "olm", "error": "OperatorCo
ndition.operators.coreos.com \"packageserver\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
time="2022-03-02T05:55:45Z" level=debug msg="skipping requeue since CSV packageserver is not in cache" namespace=olm ownee=packageserver-service-cert selflink=
1.6462005457989151e+09  ERROR   controllers.operatorcondition   Unable to find operatorcondition        {"request": "olm/packageserver", "error": "OperatorCondition.operators.coreos.com \"packageserver\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/njhale/projects/operator-framework/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
1.646200545799314e+09   ERROR   controller.operatorcondition    Reconciler error        {"reconciler group": "operators.coreos.com", "reconciler kind": "OperatorCondition", "name": "packageserver", "namespace": "olm", "error": "OperatorCo
ndition.operators.coreos.com \"packageserver\" not found"

Environment

• operator-lifecycle-manager version:

5cf7f1f4e

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:11:29Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2022-01-06T22:13:50Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: kind

Possible Solution

Don't retry OperatorCondition reconciliation when expected CSVs are not found; explicitly, when the response for a GET request is 404 NOT_FOUND.