[BUG] CRIO restart fails the kube-rbac-proxy with the following error - /usr/bin/kube-rbac-proxy: error while loading shared libraries: libresolv.so.2: cannot open shared object file: Permission denied

[aayushagr22]

We've setup Microshift 4.18 on a CIS Hardened RHEL machine using a local podman registry. When we disable nftables to allow firewall the local registry goes down and to bring it back following commands were run:

sudo systemctl restart crio
podman stop local-registry
podman start local-registry

Post this all pods are up expect the kube-rbac-proxy container in the dns-default pod. Seems like a cleanup issue from crio.

What did you expect to happen?

How to reproduce it (as minimally and precisely as possible)?

1. '...'
2. '...'

Anything else we need to know?

Environment

• MicroShift version (use microshift version):
• Hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

Relevant logs

[ggiguash]

Changing system configuration may require restarting more than one service.
Does the system heal itself after reboot?

[aayushagr22]

Hi @ggiguash, this was solved by replacing sudo systemctl restart crio with the following commands -

systemctl stop crio
crio wipe -f
systemctl start crio

[aayushagr22]

Hi,

The same issue re-occured on rebooting the server and the previous fix doesn't work.

oc logs -f -n openshift-service-ca service-ca-7674ff74cb-flrrc;oc logs -f -n openshift-ingress router-default-5c6b6bf9cb-xndhs
exec container process /usr/bin/service-ca-operator: Permission denied
/usr/bin/openshift-router: error while loading shared libraries: libresolv.so.2: cannot open shared object file: Permission denied

[ggiguash]

Permission denied errors like that are sometimes related to SELinux denials.
Can you examine SELinux events to see if it contains any clues?