Skip to content

Bug: ryuk exited instances leaking #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
shoffmeister opened this issue Mar 21, 2024 · 4 comments · Fixed by #499
Closed

Bug: ryuk exited instances leaking #489

shoffmeister opened this issue Mar 21, 2024 · 4 comments · Fixed by #499

Comments

@shoffmeister
Copy link
Contributor

shoffmeister commented Mar 21, 2024 •
edited
Loading

Describe the bug

With the new ryuk container usage in v4.1.1, dependent containers get cleaned up nicely, but the ryuk container itself remains, in exited state. The ryuk container itself should also be cleaned up.

docker ps --all will show a great many of these containers, see the very end of the bug report for some testcontainers-based pytest integration test that was running in a loop.

While docker rm -f $(docker ps --all --quiet --filter="name=testcontainers-ryuk*") will clean this up, it is still annoying to have to manage this.

To Reproduce

Any usage of testcontainers should exhibit this behaviour.

Locally, I am running this with a specialized Dockercontainer implementation which drives the official Apache Kafka 3.7.0 container image in KRaft mode (which obviates the need for Zookeeper dancing and hence cycles much much faster).

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

# Get the operating system information (on a unix os).
$ uname -a

Linux arch 6.8.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 16 Mar 2024 17:15:35 +0000 x86_64 GNU/Linux

# Get the python version.
$ python --version

Python 3.11.8

# Get the docker version and other docker information.
$ docker info

❯ docker info
Client:
 Version:    25.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.24.7
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 50
  Running: 0
  Paused: 0
  Stopped: 50
 Images: 34
 Server Version: 25.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dcf2847247e18caba8dce86522029642f60fe96b.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.1-arch1-1
 Operating System: EndeavourOS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.31GiB
 Name: arch
 ID: 9b2395c3-f6f6-4d7d-b25b-4b0d963287ae
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

# Get all python packages.
$ pip freeze

asttokens==2.4.1
backoff==2.2.1
busypie==0.5.1
certifi==2024.2.2
charset-normalizer==3.3.2
comm==0.2.2
confluent-kafka==2.3.0
confluent-kafka-stubs @ git+https://gitlab.com/shoffmeister/confluent-kafka-stubs.git@29b72e31099a89609107c541ef6f9dfef3185164
debugpy==1.8.1
decorator==5.1.1
docker==7.0.0
duckdb==0.10.1
duckdb_engine==0.11.2
executing==2.0.1
greenlet==3.0.3
idna==3.6
iniconfig==2.0.0
ipykernel==6.29.3
ipython==8.22.2
ipython-genutils==0.2.0
ipython-sql==0.5.0
jedi==0.19.1
Jinja2==3.1.3
jupysql==0.10.10
jupysql-plugin==0.4.2
jupyter_client==8.6.1
jupyter_core==5.7.2
kafka-python==2.0.2
MarkupSafe==2.1.5
matplotlib-inline==0.1.6
monotonic==1.6
mypy==1.9.0
mypy-extensions==1.0.0
nest-asyncio==1.6.0
packaging==24.0
parso==0.8.3
pexpect==4.9.0
platformdirs==4.2.0
ploomber-core==0.2.25
ploomber-extension==0.1.0
pluggy==1.4.0
posthog==3.5.0
prettytable==3.10.0
prompt-toolkit==3.0.43
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.17.2
pytest==8.1.1
pytest-dependency==0.6.0
python-dateutil==2.9.0.post0
PyYAML==6.0.1
pyzmq==25.1.2
requests==2.31.0
ruff==0.3.2
six==1.16.0
SQLAlchemy==2.0.28
sqlglot==23.0.3
sqlparse==0.4.4
stack-data==0.6.3
testcontainers==4.1.1
tornado==6.4
traitlets==5.14.2
typing_extensions==4.10.0
urllib3==2.2.1
wcwidth==0.2.13
wrapt==1.16.0

1feb2e86bec7   testcontainers/ryuk:0.5.1               "/bin/ryuk"              36 seconds ago       Exited (0) 15 seconds ago                 testcontainers-ryuk-e881adfb-5f0e-4f37-980d-cc96cb98e77b
baa9afb86caa   testcontainers/ryuk:0.5.1               "/bin/ryuk"              51 seconds ago       Exited (0) 26 seconds ago                 testcontainers-ryuk-753d4e33-b754-40ea-803a-44d5e3754f81
c3b72be91821   testcontainers/ryuk:0.5.1               "/bin/ryuk"              58 seconds ago       Exited (0) 41 seconds ago                 testcontainers-ryuk-a9a84681-93ef-4a6a-a0ec-43bc4227566e
981a0d4be1c1   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 48 seconds ago                 testcontainers-ryuk-53d2d06e-48c1-4f52-9045-5c568c67546c
031887f37bf5   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 50 seconds ago                 testcontainers-ryuk-ea29c45b-a187-4e8c-9a98-a788a7b71540
571fa2953a96   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 51 seconds ago                 testcontainers-ryuk-fb9e6553-bc15-41eb-8f23-adc66243251f
5bb48b21000e   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 52 seconds ago                 testcontainers-ryuk-aff1a002-052b-48e7-9425-dd6e0d344abf
6ab93eb9bf98   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 53 seconds ago                 testcontainers-ryuk-1b6f4b61-859f-418b-8c09-f1e3069c795d
2846d9812f13   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 55 seconds ago                 testcontainers-ryuk-7fa34e33-0616-41a5-b357-b7ffef21b90d
f8c2b8105760   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 56 seconds ago                 testcontainers-ryuk-0dd023fd-06c8-40da-8072-f200c688cc80
d0b4c7c71c58   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) 56 seconds ago                 testcontainers-ryuk-178ec9c2-85c2-47b1-a151-57ef5e6bff12
ccba9b4802c3   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) About a minute ago             testcontainers-ryuk-b1fb17dc-d87e-48d7-a34b-e3c42520b097
ef1231b2b12a   testcontainers/ryuk:0.5.1               "/bin/ryuk"              About a minute ago   Exited (0) About a minute ago             testcontainers-ryuk-97c367bc-e6c1-411d-aa97-57be26e2eb4e
921164cbb0c4   testcontainers/ryuk:0.5.1               "/bin/ryuk"              2 minutes ago        Exited (0) About a minute ago             testcontainers-ryuk-c7bdf927-b1a9-4ba2-b1ef-e5ae475a01ff
f44712b386a7   testcontainers/ryuk:0.5.1               "/bin/ryuk"              2 minutes ago        Exited (0) About a minute ago             testcontainers-ryuk-b613f053-ba7f-48a0-982d-bc737babd2b0
4391bf404514   testcontainers/ryuk:0.5.1               "/bin/ryuk"              2 minutes ago        Exited (0) 2 minutes ago                  testcontainers-ryuk-2f00dec7-519e-45b0-915f-1dbc2bb05eab
1104b643b606   testcontainers/ryuk:0.5.1               "/bin/ryuk"              2 minutes ago        Exited (0) 2 minutes ago                  testcontainers-ryuk-03761072-2a9e-439a-b0b5-a25a833ae55b
71aa1530031e   testcontainers/ryuk:0.5.1               "/bin/ryuk"              3 minutes ago        Exited (0) 2 minutes ago                  testcontainers-ryuk-9da38d7f-0e42-413f-81d5-742cd58d9820
f56a6a2e44f6   testcontainers/ryuk:0.5.1               "/bin/ryuk"              3 minutes ago        Exited (0) 2 minutes ago                  testcontainers-ryuk-b244b3c3-4c5e-4fd7-a0c9-c7b0bc5cb51a
e8172012e041   testcontainers/ryuk:0.5.1               "/bin/ryuk"              3 minutes ago        Exited (0) 3 minutes ago                  testcontainers-ryuk-d7d4b0b8-13d1-4c68-9bdb-8468cd85486c
79a37961a9cd   testcontainers/ryuk:0.5.1               "/bin/ryuk"              3 minutes ago        Exited (0) 3 minutes ago                  testcontainers-ryuk-67a2bd01-07e6-41ea-8d03-6b97e95597dd
f0e9749bf00c   testcontainers/ryuk:0.5.1               "/bin/ryuk"              4 minutes ago        Exited (0) 3 minutes ago                  testcontainers-ryuk-a590bd18-2a6e-4e85-8757-04bc76930e23
89336b1bc1d6   testcontainers/ryuk:0.5.1               "/bin/ryuk"              4 minutes ago        Exited (0) 3 minutes ago                  testcontainers-ryuk-2c5bd5c6-643a-4607-870a-9eec0abbd7e5
ea324a16bc8c   testcontainers/ryuk:0.5.1               "/bin/ryuk"              4 minutes ago        Exited (0) 4 minutes ago                  testcontainers-ryuk-cfd431c4-bb50-4bed-8375-d0fc149b5780
fcd1566778f6   testcontainers/ryuk:0.5.1               "/bin/ryuk"              4 minutes ago        Exited (0) 4 minutes ago                  testcontainers-ryuk-ee436a51-5ece-4122-b0f0-0401f9c4d1a5
fd77f7842f1f   testcontainers/ryuk:0.5.1               "/bin/ryuk"              5 minutes ago        Exited (0) 4 minutes ago                  testcontainers-ryuk-4967d460-d296-430b-b8c9-5fd343563eef
8e10e3de770f   testcontainers/ryuk:0.5.1               "/bin/ryuk"              5 minutes ago        Exited (0) 4 minutes ago                  testcontainers-ryuk-9261c4e2-e35e-4b9d-9184-13eb6d4f8e7e
1faf36416c86   testcontainers/ryuk:0.5.1               "/bin/ryuk"              5 minutes ago        Exited (0) 5 minutes ago                  testcontainers-ryuk-e6a85573-f701-4de7-b503-f11b3f3b47cf
02bb029d442d   testcontainers/ryuk:0.5.1               "/bin/ryuk"              5 minutes ago        Exited (0) 5 minutes ago                  testcontainers-ryuk-542b0790-1673-4532-a281-8ecfae0c3174
f711663ce13e   testcontainers/ryuk:0.5.1               "/bin/ryuk"              6 minutes ago        Exited (0) 5 minutes ago                  testcontainers-ryuk-f903520b-d021-4ac5-98f7-6d88c486f232
7d136c8f8532   testcontainers/ryuk:0.5.1               "/bin/ryuk"              6 minutes ago        Exited (0) 5 minutes ago                  testcontainers-ryuk-a59f1aac-6bd3-4e2a-bcde-0fdc0b2d839f
75114daca8ca   testcontainers/ryuk:0.5.1               "/bin/ryuk"              6 minutes ago        Exited (0) 6 minutes ago                  testcontainers-ryuk-89cd6f0f-245b-4499-b7d0-c59b66fed775
8a5f76e9b660   testcontainers/ryuk:0.5.1               "/bin/ryuk"              6 minutes ago        Exited (0) 6 minutes ago                  testcontainers-ryuk-24a5875a-7c71-4c9d-8f92-b9cc4708ee25
a8ab0510105e   testcontainers/ryuk:0.5.1               "/bin/ryuk"              7 minutes ago        Exited (0) 6 minutes ago                  testcontainers-ryuk-14eb7d3c-1552-4db5-89ee-4669ff94e353
9757e6699598   testcontainers/ryuk:0.5.1               "/bin/ryuk"              7 minutes ago        Exited (0) 6 minutes ago                  testcontainers-ryuk-e69fdc48-bda7-40cf-9ff7-ef0352648b5a
8ca4efc76b39   testcontainers/ryuk:0.5.1               "/bin/ryuk"              7 minutes ago        Exited (0) 7 minutes ago                  testcontainers-ryuk-be6c05d8-c8eb-4d65-abaf-99d8882c4aa8
b001249b35d8   testcontainers/ryuk:0.5.1               "/bin/ryuk"              7 minutes ago        Exited (0) 7 minutes ago                  testcontainers-ryuk-85cacd09-d414-4b17-9595-65413c17a0a8
f3038dab3c61   testcontainers/ryuk:0.5.1               "/bin/ryuk"              8 minutes ago        Exited (0) 7 minutes ago                  testcontainers-ryuk-358bc1e5-b278-451c-b7f0-bfe393656634
7c6225374c36   testcontainers/ryuk:0.5.1               "/bin/ryuk"              8 minutes ago        Exited (0) 7 minutes ago                  testcontainers-ryuk-ece119eb-09fd-4cb8-b992-ba14ef6a4fa3
dad0e439e20e   testcontainers/ryuk:0.5.1               "/bin/ryuk"              8 minutes ago        Exited (0) 8 minutes ago                  testcontainers-ryuk-d9233fe8-a132-4ae2-bf62-6e003812b727
517f1769e141   testcontainers/ryuk:0.5.1               "/bin/ryuk"              9 minutes ago        Exited (0) 8 minutes ago                  testcontainers-ryuk-b329a687-9e63-4abe-aa98-5421a4bcc496
2dd542698248   testcontainers/ryuk:0.5.1               "/bin/ryuk"              9 minutes ago        Exited (0) 8 minutes ago                  testcontainers-ryuk-4c861aaa-baf5-4599-8b1b-c67c99714f95
aa8e63e3042a   testcontainers/ryuk:0.5.1               "/bin/ryuk"              9 minutes ago        Exited (0) 9 minutes ago                  testcontainers-ryuk-c589f726-e0cf-4faa-b453-037cd642ca24
alexanderankin added a commit that referenced this issue Mar 23, 2024
@alexanderankin
suggest a fix for #489
cf78b03
@alexanderankin alexanderankin mentioned this issue Mar 23, 2024
Closed
@alexanderankin
Copy link
Member

alexanderankin commented Mar 23, 2024 •
edited
Loading

@shoffmeister can you test with https://pypi.org/project/testcontainers/4.2.0rc1/ ? thanks! nevermind, this approach will not work yet.

@alexanderankin
Copy link
Member

https://pypi.org/project/testcontainers/4.2.0rc2/ is up, tested a bit by me, maybe fixes the issue?

@shoffmeister
Copy link
Contributor Author

@alexanderankin Many thanks for the very quick reaction!

4.2.0.rc2 has survived a few minutes of while true; do pytest; done without leaking the ryuk container.

There is one oddity, though: Ctrl+C does not abort the tests. To reproduce

  • pytest (just a simple one, no need for the infinite loop)
  • run a test suite which uses (my) Kafka container
  • Ctrl+C

//exp: test run aborts
//act: test run stops (with ryuk + Kafka container gone) and appears to hang dead

Right now I do not know how to minimally reproduce this, what the root cause of that is, and whether this is a regression - it may simply be due to timeouts that are too high (ryuk, my container).

I believe that the current state is strictly better than the previous state, so I'd suggest proceeding with the current state for now.

@alexanderankin alexanderankin mentioned this issue Mar 24, 2024
Merged
alexanderankin added a commit that referenced this issue Mar 24, 2024
@alexanderankin @shoffmeister
fix(core): use auto_remove=True with reaper instance (#499)
274a400 
fix #489 , supercede #491, #498

Co-authored-by: Stefan Hoffmeister <[email protected]>
@alexanderankin
Copy link
Member

@shoffmeister - after a day of experimenting i realized that the shutdown hooks are not necessary - and actually just how necessary the ryuk is in practice. im releasing 4.2.0 right now with only the necessary parts of the above fix, and not the unnecessary ones. thanks for reporting and helping to test!

bearrito pushed a commit to bearrito/testcontainers-python that referenced this issue Mar 30, 2024
@alexanderankin @shoffmeister
fix(core): use auto_remove=True with reaper instance (testcontainers#499
135866b 
)

fix testcontainers#489 , supercede testcontainers#491, testcontainers#498

Co-authored-by: Stefan Hoffmeister <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(core): use auto_remove=True with reaper instance testcontainers/testcontainers-python
2 participants
@shoffmeister @alexanderankin