image-registry Available: The deployment does not exist.. | Unable to apply 4.16.15: the cluster operator image-registry is not available

[n00bsi]

[provide a description of the issue]

Version

[provide output of the openshift version or oc version command]

$ oc version
Client Version: 4.15.11
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.16.14
Kubernetes Version: v1.29.8+f10c92d

Steps To Reproduce

1. start Update from 4.16.14 to 4.16.15
2. update hang at 88%

Current Result

Update hat at 88%

Available: The deployment does not exist NodeCADaemonAvailable: The daemon set node-ca has available replicas ImagePrunerAvailable: Pruner CronJob has been created

$ oc describe pod -n openshift-image-registry node-ca-5c6gg | grep Node
Node-Selectors:              kubernetes.io/os=linux
  Warning  NodeNotReady  100m (x3 over 5h21m)  node-controller  Node i
```s not ready

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.16.14 True True 6d Unable to apply 4.16.15: the cluster operator image-registry is not available

$ oc get clusteroperator image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry False True True 26m Available: The deployment does not exist...

$ oc get pvc
...
...
ocs4registry Bound pvc-38960e2f-4c6b-450d-a5fe-c1a26714e496 1Gi RWX longhorn 162
...

How to fix this ?

$ oc get pods -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
cluster-image-registry-operator-7c87776c4c-csz22 1/1 Running 0 38m
node-ca-5c6gg 1/1 Running 0 38m
node-ca-c492l 1/1 Running 0 38m
node-ca-crzlc 1/1 Running 0 156m
node-ca-dskf6 1/1 Running 0 38m
node-ca-mpwjb 1/1 Running 0 38m
node-ca-xmjbp 1/1 Running 0 38m

Output of: `oc edit configs.imageregistry.operator.openshift.io -o yaml`

see this attach
[image_reg.yaml.log](https://github.com/user-attachments/files/17459954/image_reg.yaml.log)


##### Expected Result

Update go to the end

##### Additional Information
[try to run `$ oc adm diagnostics` (or `oadm diagnostics`) command if possible]
[if you are reporting issue related to builds, provide build logs with `BUILD_LOGLEVEL=5`]
[consider attaching output of the `$ oc get all -o json -n <namespace>` command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

[n00bsi]

Found some things:

oc describe clusteroperator/machine-config

oc delete pod node-ca-8566c -n openshift-image-registry
and all other node-* pods

oc get pods -n openshift-machine-config-operator 
oc logs -f -n openshift-machine-config-operator machine-config-controller-645db999c6-xjsqs -c machine-config-controller

oc adm drain node1.domain.tld --ignore-daemonsets --force --delete-emptydir-data

https://www.neteye-blog.com/2023/08/debug-and-workarounds-for-a-stuck-update-on-openshift-4-13-6/

https://access.redhat.com/solutions/5317441

https://access.redhat.com/solutions/5598401

Now all Nodes have the same level:

Red Hat Enterprise Linux CoreOS 416.94.202409191851-0

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.15   True        False         112m    Cluster version is 4.16.15

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[restlingclone]

I have also got the same error while updating SNO cluster from 4.17.14 to 4.17.18 What i have done is edit the image-registry added entry under spec section managementState: Removed and also removed httpSecret and it got resolved.
Few things to remeber when we have storage in build inside hardware adn ready to use always managementState: Removed under spec section.

Error while Cluster opertor update:

Administrator-->Namespaces search image-registry

Herer is my old yaml file entry:
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
finalizers:

• imageregistry.operator.openshift.io/finalizer
  generation: 4
  name: cluster
  uid: f3955a86-33d4-4316-9b20-0747a877a
  spec:
  httpSecret: 2656d471bffd0a7ab8083f5931968ac701f631ecd62d79b9b2361ba4fc192fbb2b8f52950989a716fa9f873a8f00733ef3cd985f31e67a0dcd5f8f5c506e540c
  logLevel: Normal
  observedConfig: null
  operatorLogLevel: Normal
  proxy: {}
  replicas: 1
  requests:
  read:
  maxWaitInQueue: 0s
  write:
  maxWaitInQueue: 0s
  rolloutStrategy: RollingUpdate
  storage: {}
  unsupportedConfigOverrides: null

New yaml file entry:
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
finalizers:

• imageregistry.operator.openshift.io/finalizer
  generation: 1
  name: cluster
  uid: 3c33eb97-3e48-4343-b0e7-20b3496y
  spec:
  logLevel: Normal
  managementState: Removed
  observedConfig: null
  operatorLogLevel: Normal
  proxy: {}
  replicas: 1
  requests:
  read:
  maxWaitInQueue: 0s
  write:
  maxWaitInQueue: 0s
  rolloutStrategy: RollingUpdate
  storage: {}
  unsupportedConfigOverrides: null

Here are the steps to do:-
1] Bakcup of same yaml file
$ oc get config.imageregistry.operator.openshift.io/cluster -o yaml > myconfig.imageregistry.yaml

2] edit the file and save it.
$ oc edit config.imageregistry.operator.openshift.io/cluster

...
sped:
managementState: Managed
...

It will automatic restart and show us that image-registry has been successfully updated.

After successfully update this how entry inside Administrator-->Namespaces search image-registry

This is how we resolve for SNO cluster. If we are not using any additional public storage or external storage with cluster.