Skip to content

cAdvisor Fails to Retrieve Metrics for Podman Container Managed by systemd #3648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dmennecke1 opened this issue Jan 24, 2025 · 4 comments · May be fixed by #3684
Open

cAdvisor Fails to Retrieve Metrics for Podman Container Managed by systemd #3648

dmennecke1 opened this issue Jan 24, 2025 · 4 comments · May be fixed by #3684

Comments

@dmennecke1
Copy link

Issue Description

When running an NGINX container manually using the following command:

podman run --name=nginx -e=CONTAINER_NAME=nginx -p=8081:80 --restart=always -d docker.io/nginx

cAdvisor is able to successfully retrieve and display container metrics:

container_cpu_load_average_10s{container_label_BACKEND_URL="",container_label_ERROR_FILE="",container_label_io_buildah_version="",container_label_maintainer="NGINX Docker Maintainers <docker [email protected]>",container_label_org_opencontainers_image_ref_name="",container_label_org_opencontainers_image_version="",id="/user.slice/user-1001.slice/[email protected]/user.slice/libpod-76e7df60cd24ecdabca6f5ef54b2ba87bbca060adeb0018ddb2c0017f18c50db.scope",image="docker.io/library/nginx:latest",name="nginx"} 0 1737716804296

However, when starting the container as a systemd service (based on the Quadlet feature), cAdvisor fails to retrieve metrics.

Systemd Service Configuration

The container is started using the following systemd service unit:

# Automatically generated by /usr/local/lib/systemd/user-generators/podman-user-generator
#
[X-Container]
ContainerName=nginx
Image=docker.io/nginx
PublishPort=8081:80

[Install]
WantedBy=default.target

[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
SourcePath=/etc/containers/systemd/users/1001/nginx.container
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/local/bin/podman run --name nginx --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --publish 8081:80 --env CONTAINER_NAME=nginx  docker.io/nginx

Observed Errors

When the container is started via systemd, the following cAdvisor errors appear in the logs:

E0124 12:16:27.057400       1 manager.go:1116] Failed to create existing container: /user.slice/user-1001.slice/[email protected]/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

W0124 12:16:27.109060       1 manager.go:1169] Failed to process watch event {EventType:0 Name:/user.slice/user-1001.slice/[email protected]/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d WatchSource:0}: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

Expected Behavior

cAdvisor should return metrics for the container when it is managed by systemd, similar to the manual Podman execution.

Actual Behavior

cAdvisor fails to collect container metrics and logs errors related to RW layer ID determination.
@w00di
Copy link

w00di commented Feb 17, 2025

I've also been troubleshooting this and have yet to find a solid work around.

@w00di
Copy link

w00di commented Feb 17, 2025

prometheus-podman-exporter has been a temporary replacement for just monitoring the containers.

@mstinsky
Copy link

mstinsky commented Apr 3, 2025

I have the same issue while using Podman with Quadlet files.
I've tried multiple things but can't seem to get it to work.

Did any of you find a solution?

This was referenced May 10, 2025
Open
Open
@savely-krasovsky
Copy link

@mstinsky I've prepared a fix, it's linked above. I tested it and now it properly shows all the containers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: support podman volatile containers savely-krasovsky/cadvisor
4 participants
@savely-krasovsky @mstinsky @w00di @dmennecke1