Skip to content

exponentially increasing libnvidia-allocator mount points on host #697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanQuigley opened this issue Sep 17, 2024 · 3 comments
Closed
Assignees

Comments

@BryanQuigley
Copy link

This appears to be the same root issue as #660 or #663 but wanted to detail the impact we are seeing from it.
Is there any timeline for the backported fix to be released?

Our symptoms

When mounting with a specific container image and a shared mount we get it creating an exponentially increasing amount of mounts on the host.

It actually creates 2^x - 1 number of mount points, so 1, 3 ... .255.. 32767. Where x is the number of containers causing the issue have run.

When it gets to the higher numbers it starts causing significant system issues due to the very high number of mounts.

The mount point is basically the same as from the other issue - copying here for searchability (although we are on nvidia 560)

/dev/md0 on /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.54.15 type ext4 (rw,relatime,nodelalloc,errors=remount-ro)```
@BryanQuigley
Copy link
Author

@elezar was this what you were seeing too?

@elezar
Copy link
Member

elezar commented Sep 18, 2024

@BryanQuigley yes, this is the known issue and is triggered when NVIDIA_DRIVER_CAPABILITIES includes graphics or display. It should only be triggered if a container is started with a bidirectional mount propagation.

We have a fix in place and this will be included in the next patch release. For the time being, please downgrade to a v1.15.x version if possible.

@elezar elezar self-assigned this Sep 18, 2024
@BryanQuigley
Copy link
Author

Thanks! I see it was released in 1.16.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants