CAdvisor with podman does not discover subcontainers

[philippfriese]

Problem

Similar to #3245, running cAdvisor, build from commit bf2a7fe (latest on master at time of writing), does not discover subcontainers.

Context

cAdvisor version bf2a7fe is run on a "rootless"/regular podman. cAdvisor was built using buildah version 1.32.0. The system is an ArchLinux on kernel 6.5.8-arch1-1.

The command used to invoke cAdvisor is:

-> % podman run \
  --rm \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/home/pfriese/.local/share/containers/:/var/lib/containers:ro \
  --volume=/etc/machine-id:/etc/machine-id:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  cadvisor:latest \
  --podman="unix:///var/run/user/1000/podman/podman.sock" \
  --v 100

Two other containers were launched, which do appear in podman ps:

CONTAINER ID  IMAGE                               COMMAND               CREATED            STATUS         PORTS                   NAMES
b7c1a26b0a58  docker.io/library/cassandra:latest  cassandra -f          About an hour ago  Up 42 minutes                          cassandra
9a27f66aa9a2  docker.io/library/cassandra:latest  cqlsh cassandra -...  12 minutes ago     Up 12 minutes                          interesting_almeida

Using regular root Docker version 24.0.6 works out of the box.

Further Attempts

Trying to run cAdvisor in rootful podman, i.e. via sudo podman, or by explicitly mapping UID/GID 0, does not change the observed behaviour.

Further Observations

Curiously, if I use docker to launch cAdvisor, pass the --podman="unix:///var/run/user/1000/podman/podman.sock flag to it, and simultaneously run containers using podman, then these appear as subcontainers in cAdvisor!
Inversely, if I run cAdvisor via podman and simultaneously run containers using docker, these do not show up as subcontainers in cAdvisor.

Comparing the logs between podman and docker cAdvisor, I notice that docker emits significantly more messages and explicitly acknowledges adding subcontainers.

Podman

-> % podman run [...]

[...]
I1026 14:40:39.725603  188557 manager.go:234] Version: {KernelVersion:6.5.8-arch1-1 ContainerOsVersion:Alpine Linux v3.16 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.47.0.126+bf2a7fee4170e4 CadvisorRevision:bf2a7fee}
I1026 14:40:40.013277  188557 plugin.go:89] Registering Podman factory
I1026 14:40:40.245593  188557 factory.go:221] Registration of the podman container factory successfully
I1026 14:40:40.245603  188557 factory.go:55] Registering systemd factory
I1026 14:40:40.245606  188557 factory.go:221] Registration of the systemd container factory successfully
I1026 14:40:40.245953  188557 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I1026 14:40:40.245985  188557 factory.go:219] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: permission denied
I1026 14:40:40.246025  188557 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I1026 14:40:40.246080  188557 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied
I1026 14:40:40.246098  188557 factory.go:103] Registering Raw factory
I1026 14:40:40.246107  188557 manager.go:1197] Started watching for new ooms in manager
W1026 14:40:40.246120  188557 manager.go:307] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: operation not permitted
I1026 14:40:40.246127  188557 factory.go:279] Factory "podman" was unable to handle container "/"
I1026 14:40:40.246132  188557 factory.go:45] / not handled by systemd handler
I1026 14:40:40.246135  188557 factory.go:279] Factory "systemd" was unable to handle container "/"
I1026 14:40:40.246140  188557 factory.go:275] Using factory "raw" for container "/"
I1026 14:40:40.246298  188557 manager.go:982] Added container: "/" (aliases: [], namespace: "")
I1026 14:40:40.246388  188557 handler.go:325] Added event &{/ 2023-10-26 14:40:34.687104394 +0000 UTC containerCreation {<nil>}}
I1026 14:40:40.246403  188557 manager.go:320] Starting recovery of all containers
I1026 14:40:40.246421  188557 container.go:527] Start housekeeping for container "/"
I1026 14:40:40.246440  188557 manager.go:325] Recovery completed
I1026 14:40:40.246865  188557 cadvisor.go:176] Starting cAdvisor version: v0.47.0.126+bf2a7fee4170e4-bf2a7fee on port 8080

Docker

-> % docker run [...]
[...]
I1026 14:41:20.629965       1 manager.go:234] Version: {KernelVersion:6.5.8-arch1-1 ContainerOsVersion:Alpine Linux v3.16 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.47.0.126+bf2a7fee4170e4 CadvisorRevision:bf2a7fee}
I1026 14:41:20.630326       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I1026 14:41:20.631361       1 factory.go:145] Registering containerd factory
I1026 14:41:20.631369       1 factory.go:221] Registration of the containerd container factory successfully
I1026 14:41:20.631420       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I1026 14:41:20.638315       1 factory.go:352] Registering Docker factory
I1026 14:41:20.638323       1 factory.go:221] Registration of the docker container factory successfully
I1026 14:41:20.918026       1 plugin.go:89] Registering Podman factory
I1026 14:41:21.155317       1 factory.go:221] Registration of the podman container factory successfully
I1026 14:41:21.155330       1 factory.go:55] Registering systemd factory
I1026 14:41:21.155332       1 factory.go:221] Registration of the systemd container factory successfully
I1026 14:41:21.155351       1 factory.go:103] Registering Raw factory
I1026 14:41:21.155359       1 manager.go:1197] Started watching for new ooms in manager
I1026 14:41:21.155398       1 factory.go:279] Factory "containerd" was unable to handle container "/"
I1026 14:41:21.155410       1 factory.go:279] Factory "docker" was unable to handle container "/"
I1026 14:41:21.155416       1 factory.go:279] Factory "podman" was unable to handle container "/"
I1026 14:41:21.155423       1 factory.go:45] / not handled by systemd handler
I1026 14:41:21.155425       1 factory.go:279] Factory "systemd" was unable to handle container "/"
I1026 14:41:21.155430       1 factory.go:275] Using factory "raw" for container "/"
I1026 14:41:21.155533       1 manager.go:982] Added container: "/" (aliases: [], namespace: "")
I1026 14:41:21.155591       1 handler.go:325] Added event &{/ 2023-10-26 13:38:49.938433091 +0000 UTC containerCreation {<nil>}}
I1026 14:41:21.155609       1 manager.go:320] Starting recovery of all containers
I1026 14:41:21.155636       1 container.go:527] Start housekeeping for container "/"
I1026 14:41:21.157583       1 factory.go:279] Factory "containerd" was unable to handle container "/system.slice/systemd-journald.service"
I1026 14:41:21.157593       1 factory.go:279] Factory "docker" was unable to handle container "/system.slice/systemd-journald.service"
I1026 14:41:21.157601       1 factory.go:279] Factory "podman" was unable to handle container "/system.slice/systemd-journald.service"
I1026 14:41:21.157608       1 factory.go:45] /system.slice/systemd-journald.service not handled by systemd handler
I1026 14:41:21.157611       1 factory.go:279] Factory "systemd" was unable to handle container "/system.slice/systemd-journald.service"
I1026 14:41:21.157616       1 factory.go:275] Using factory "raw" for container "/system.slice/systemd-journald.service"
[...]
I1026 14:41:21.179376       1 manager.go:982] Added container: "/system.slice/docker-88c013046b35b9b09c5e05e9e967ad4a02d881235959812a2a40532362c356af.scope" (aliases: [cadvisor 88c013046b35b9b09c5e05e9e967ad4a02d881235959812a2a40532362c356af], namespace: "docker")
[...]
I1026 14:41:21.199590       1 cadvisor.go:176] Starting cAdvisor version: v0.47.0.126+bf2a7fee4170e4-bf2a7fee on port 8080
[...]

The command used to invoke cAdvisor via docker is equivalent to the podman commant except replacing podman with docker.

[Crapshit]

Perhaps I'm having the same issue like you @philippfriese.
I noticed after creating cAdvisor container multiple times with the same Podman run command, that it is sometimes possible to gather metrics from other container's.
And often not. But I changed nothing.... So weird.
Do you have the same issue?

[Cub0n]

Can you please test it with cAdvisor 0.49.1.
On my RaspberryPI every Container is found.

[Crapshit]

I used v. 0.49.1 on RHEL 8 with up2date OS & Podman version.

[Crapshit]

Hi all, Hi @philippfriese ,

I did some digging and I found the following on GitHub gists:

https://gist.github.com/mikroskeem/c6b6b4728dd8a45a2db212b3565ddb81?permalink_comment_id=4766279#gistcomment-4766279

With the following podman run command I got it working on Ubuntu 22.04:

podman run -d --name cadvisor \
--volume /:/rootfs:ro \
--volume /dev/disk/:/dev/disk:ro \
--volume /etc/machine-id:/etc/machine-id:ro \
--volume /sys:/sys:ro \
--volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume /var/lib/containers:/var/lib/containers:ro \
--volume /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro \
--volume /var/run:/var/run:rw \
--privileged \
-p 127.0.0.1:8080:8080 \
gcr.io/cadvisor/cadvisor:v0.49.1

On RHEL 9 it failed because of:

--volume /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro \

So I removed it and it seems to work now:

podman run -d --name cadvisor \
--volume /:/rootfs:ro \
--volume /dev/disk/:/dev/disk:ro \
--volume /etc/machine-id:/etc/machine-id:ro \
--volume /sys:/sys:ro \
--volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume /var/lib/containers:/var/lib/containers:ro \
--volume /var/run:/var/run:rw \
--privileged \
-p 127.0.0.1:8080:8080 \
gcr.io/cadvisor/cadvisor:v0.49.1

Additionally I added the following...:

--podman=unix:///run/podman/podman.sock --docker=unix://

.... to avoid any issues because of podman-docker package:
#3554

Thank you @Creatone , @chopeen, @rushilenekar20

[usbport11]

Good day.

[root@prometh system]# dnf install docker

[root@prometh system]# dnf list installed | grep docker
podman-docker.noarch                     4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream

[root@prometh system]# dnf list installed | grep podman
podman.x86_64                            4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream
podman-catatonit.x86_64                  4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream
podman-docker.noarch                     4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream
podman-gvproxy.x86_64                    4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream
podman-plugins.x86_64                    4:4.9.4-13.0.1.module+el8.10.0+90412+9b361f34 @ol8_appstream

[root@prometh system]# docker --version
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
podman version 4.9.4-rhel

wget https://github.com/google/cadvisor/releases/download/v0.49.1/cadvisor-v0.49.1-linux-amd64
mv cadvisor-v0.49.1-linux-amd64 /usr/bin/cadvisor
chmod +x /usr/bin/cadvisor

find / -name docker.sock
/run/docker.sock

vi /usr/lib/systemd/system/cadvisor.service
[Unit]
Description=cAdvisor
Wants=network-online.target
After=network-online.target
[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/cadvisor --podman=unix:///run/docker.sock
[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start cadvisor

http://localhost:8080/podman/
failed to get podman info: response not present: Get "http://d/v1.0.0/info": dial unix /run/docker.sock: connect: no such file or directory
http://localhost:8080/docker/
failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

ExecStart=/usr/bin/cadvisor --podman=/run/docker.sock

systemctl daemon-reload
systemctl start cadvisor

http://localhost:8080/podman/
failed to get podman info: couldn't get podman client
http://localhost:8080/docker/
failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

find / -name podman.sock
/run/podman/podman.sock

ExecStart=/usr/bin/cadvisor --podman=unix:///run/podman/podman.sock --docker=unix://

systemctl daemon-reload
systemctl start cadvisor

http://localhost:8080/podman/
Cadvisor saw podman container. Seems good.