Skip to content

Commit 44fb868

Browse files
Merge branch 'main' into network
2 parents b523301 + 13742a5 commit 44fb868

File tree

20 files changed

+1298
-647
lines changed

20 files changed

+1298
-647
lines changed

.github/.release-please-manifest.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
{
2-
".": "4.0.1"
2+
".": "4.1.0"
33
}

.github/release-please-config.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
{
22
"release-type": "python",
3-
"bootstrap-sha": "dcb4f6842cbfe6e880a77b0d4aabb3f396c6dc19",
43
"packages": {
54
".": {
65
"package-name": "testcontainers"

CHANGELOG.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,19 @@
11
# Changelog
22

3+
## [4.1.0](https://github.com/testcontainers/testcontainers-python/compare/testcontainers-v4.0.1...testcontainers-v4.1.0) (2024-03-19)
4+
5+
6+
### Features
7+
8+
* **reliability:** integrate the ryuk container for better container cleanup ([#314](https://github.com/testcontainers/testcontainers-python/issues/314)) ([d019874](https://github.com/testcontainers/testcontainers-python/commit/d0198744c3bdc97a7fe41879b54acb2f5ee7becb))
9+
10+
11+
### Bug Fixes
12+
13+
* changelog after release-please ([#469](https://github.com/testcontainers/testcontainers-python/issues/469)) ([dcb4f68](https://github.com/testcontainers/testcontainers-python/commit/dcb4f6842cbfe6e880a77b0d4aabb3f396c6dc19))
14+
* **configuration:** strip whitespaces when reading .testcontainers.properties ([#474](https://github.com/testcontainers/testcontainers-python/issues/474)) ([ade144e](https://github.com/testcontainers/testcontainers-python/commit/ade144ee2888d4044ac0c1dc627f47da92789e06))
15+
* try to fix release-please by setting a bootstrap sha ([#472](https://github.com/testcontainers/testcontainers-python/issues/472)) ([ca65a91](https://github.com/testcontainers/testcontainers-python/commit/ca65a916b719168c57c174d2af77d45fd026ec05))
16+
317
## [4.0.1](https://github.com/testcontainers/testcontainers-python/compare/testcontainers-v4.0.0...testcontainers-v4.0.1) (2024-03-11)
418

519

INDEX.rst

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ testcontainers-python facilitates the use of Docker containers for functional an
2020
modules/clickhouse/README
2121
modules/elasticsearch/README
2222
modules/google/README
23+
modules/influxdb/README
2324
modules/kafka/README
2425
modules/keycloak/README
2526
modules/localstack/README
@@ -92,6 +93,21 @@ When trying to launch a testcontainer from within a Docker container, e.g., in c
9293
1. The container has to provide a docker client installation. Either use an image that has docker pre-installed (e.g. the `official docker images <https://hub.docker.com/_/docker>`_) or install the client from within the `Dockerfile` specification.
9394
2. The container has to have access to the docker daemon which can be achieved by mounting `/var/run/docker.sock` or setting the `DOCKER_HOST` environment variable as part of your `docker run` command.
9495

96+
Configuration
97+
-------------
98+
99+
+-------------------------------------------+-------------------------------+------------------------------------------+
100+
| Env Variable | Example | Description |
101+
+===========================================+===============================+==========================================+
102+
| ``TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE`` | ``/var/run/docker.sock`` | Path to Docker's socket used by ryuk |
103+
+-------------------------------------------+-------------------------------+------------------------------------------+
104+
| ``TESTCONTAINERS_RYUK_PRIVILEGED`` | ``false`` | Run ryuk as a privileged container |
105+
+-------------------------------------------+-------------------------------+------------------------------------------+
106+
| ``TESTCONTAINERS_RYUK_DISABLED`` | ``false`` | Disable ryuk |
107+
+-------------------------------------------+-------------------------------+------------------------------------------+
108+
| ``RYUK_CONTAINER_IMAGE`` | ``testcontainers/ryuk:0.5.1`` | Custom image for ryuk |
109+
+-------------------------------------------+-------------------------------+------------------------------------------+
110+
95111
Development and Contributing
96112
----------------------------
97113

README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,12 @@ For more information, see [the docs][readthedocs].
2222
```
2323

2424
The snippet above will spin up a postgres database in a container. The `get_connection_url()` convenience method returns a `sqlalchemy` compatible url we use to connect to the database and retrieve the database version.
25+
26+
## Configuration
27+
28+
| Env Variable | Example | Description |
29+
| ----------------------------------------- | ----------------------------- | ---------------------------------------- |
30+
| `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE` | `/var/run/docker.sock` | Path to Docker's socket used by ryuk |
31+
| `TESTCONTAINERS_RYUK_PRIVILEGED` | `false` | Run ryuk as a privileged container |
32+
| `TESTCONTAINERS_RYUK_DISABLED` | `false` | Disable ryuk |
33+
| `RYUK_CONTAINER_IMAGE` | `testcontainers/ryuk:0.5.1` | Custom image for ryuk |

core/testcontainers/core/config.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,8 @@
33
MAX_TRIES = int(environ.get("TC_MAX_TRIES", 120))
44
SLEEP_TIME = int(environ.get("TC_POOLING_INTERVAL", 1))
55
TIMEOUT = MAX_TRIES * SLEEP_TIME
6+
7+
RYUK_IMAGE: str = environ.get("RYUK_CONTAINER_IMAGE", "testcontainers/ryuk:0.5.1")
8+
RYUK_PRIVILEGED: bool = environ.get("TESTCONTAINERS_RYUK_PRIVILEGED", "false") == "true"
9+
RYUK_DISABLED: bool = environ.get("TESTCONTAINERS_RYUK_DISABLED", "false") == "true"
10+
RYUK_DOCKER_SOCKET: str = environ.get("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE", "/var/run/docker.sock")

core/testcontainers/core/container.py

Lines changed: 72 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
1-
import contextlib
21
from platform import system
3-
from typing import Optional
4-
5-
from docker.models.containers import Container
2+
from socket import socket
3+
from typing import TYPE_CHECKING, Optional
64

5+
from testcontainers.core.config import RYUK_DISABLED, RYUK_DOCKER_SOCKET, RYUK_IMAGE, RYUK_PRIVILEGED
76
from testcontainers.core.docker_client import DockerClient
87
from testcontainers.core.exceptions import ContainerStartException
8+
from testcontainers.core.labels import LABEL_SESSION_ID, SESSION_ID
99
from testcontainers.core.network import Network
1010
from testcontainers.core.utils import inside_container, is_arm, setup_logger
11-
from testcontainers.core.waiting_utils import wait_container_is_ready
11+
from testcontainers.core.waiting_utils import wait_container_is_ready, wait_for_logs
12+
13+
if TYPE_CHECKING:
14+
from docker.models.containers import Container
1215

1316
logger = setup_logger(__name__)
1417

@@ -26,7 +29,12 @@ class DockerContainer:
2629
... delay = wait_for_logs(container, "Hello from Docker!")
2730
"""
2831

29-
def __init__(self, image: str, docker_client_kw: Optional[dict] = None, **kwargs) -> None:
32+
def __init__(
33+
self,
34+
image: str,
35+
docker_client_kw: Optional[dict] = None,
36+
**kwargs,
37+
) -> None:
3038
self.env = {}
3139
self.ports = {}
3240
self.volumes = {}
@@ -69,7 +77,10 @@ def maybe_emulate_amd64(self) -> "DockerContainer":
6977
return self.with_kwargs(platform="linux/amd64")
7078
return self
7179

72-
def start(self) -> "DockerContainer":
80+
def start(self):
81+
if not RYUK_DISABLED and self.image != RYUK_IMAGE:
82+
logger.debug("Creating Ryuk container")
83+
Reaper.get_instance()
7384
logger.info("Pulling image %s", self.image)
7485
docker_client = self.get_docker_client()
7586
self._container = docker_client.run(
@@ -80,7 +91,7 @@ def start(self) -> "DockerContainer":
8091
ports=self.ports,
8192
name=self._name,
8293
volumes=self.volumes,
83-
**self._kwargs
94+
**self._kwargs,
8495
)
8596
logger.info("Container started: %s", self._container.short_id)
8697
if self._network:
@@ -91,21 +102,12 @@ def stop(self, force=True, delete_volume=True) -> None:
91102
self._container.remove(force=force, v=delete_volume)
92103
self.get_docker_client().client.close()
93104

94-
def __enter__(self) -> "DockerContainer":
105+
def __enter__(self):
95106
return self.start()
96107

97108
def __exit__(self, exc_type, exc_val, exc_tb) -> None:
98109
self.stop()
99110

100-
def __del__(self) -> None:
101-
"""
102-
__del__ runs when Python attempts to garbage collect the object.
103-
In case of leaky test design, we still attempt to clean up the container.
104-
"""
105-
with contextlib.suppress(Exception):
106-
if self._container is not None:
107-
self.stop()
108-
109111
def get_container_host_ip(self) -> str:
110112
# infer from docker host
111113
host = self.get_docker_client().host()
@@ -153,7 +155,7 @@ def with_volume_mapping(self, host: str, container: str, mode: str = "ro") -> "D
153155
self.volumes[host] = mapping
154156
return self
155157

156-
def get_wrapped_container(self) -> Container:
158+
def get_wrapped_container(self) -> "Container":
157159
return self._container
158160

159161
def get_docker_client(self) -> DockerClient:
@@ -168,3 +170,54 @@ def exec(self, command) -> tuple[int, str]:
168170
if not self._container:
169171
raise ContainerStartException("Container should be started before executing a command")
170172
return self._container.exec_run(command)
173+
174+
175+
class Reaper:
176+
_instance: "Optional[Reaper]" = None
177+
_container: Optional[DockerContainer] = None
178+
_socket: Optional[socket] = None
179+
180+
@classmethod
181+
def get_instance(cls) -> "Reaper":
182+
if not Reaper._instance:
183+
Reaper._instance = Reaper._create_instance()
184+
185+
return Reaper._instance
186+
187+
@classmethod
188+
def delete_instance(cls) -> None:
189+
if Reaper._socket is not None:
190+
Reaper._socket.close()
191+
Reaper._socket = None
192+
193+
if Reaper._container is not None:
194+
Reaper._container.stop()
195+
Reaper._container = None
196+
197+
if Reaper._instance is not None:
198+
Reaper._instance = None
199+
200+
@classmethod
201+
def _create_instance(cls) -> "Reaper":
202+
logger.debug(f"Creating new Reaper for session: {SESSION_ID}")
203+
204+
Reaper._container = (
205+
DockerContainer(RYUK_IMAGE)
206+
.with_name(f"testcontainers-ryuk-{SESSION_ID}")
207+
.with_exposed_ports(8080)
208+
.with_volume_mapping(RYUK_DOCKER_SOCKET, "/var/run/docker.sock", "rw")
209+
.with_kwargs(privileged=RYUK_PRIVILEGED)
210+
.start()
211+
)
212+
wait_for_logs(Reaper._container, r".* Started!")
213+
214+
container_host = Reaper._container.get_container_host_ip()
215+
container_port = int(Reaper._container.get_exposed_port(8080))
216+
217+
Reaper._socket = socket()
218+
Reaper._socket.connect((container_host, container_port))
219+
Reaper._socket.send(f"label={LABEL_SESSION_ID}={SESSION_ID}\r\n".encode())
220+
221+
Reaper._instance = Reaper()
222+
223+
return Reaper._instance

core/testcontainers/core/docker_client.py

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
1111
# License for the specific language governing permissions and limitations
1212
# under the License.
13-
import atexit
1413
import functools as ft
1514
import os
1615
import urllib
@@ -19,10 +18,10 @@
1918
from typing import Optional, Union
2019

2120
import docker
22-
from docker.errors import NotFound
2321
from docker.models.containers import Container, ContainerCollection
2422

25-
from .utils import default_gateway_ip, inside_container, setup_logger
23+
from testcontainers.core.labels import SESSION_ID, create_labels
24+
from testcontainers.core.utils import default_gateway_ip, inside_container, setup_logger
2625

2726
LOGGER = setup_logger(__name__)
2827
TC_FILE = ".testcontainers.properties"
@@ -42,6 +41,7 @@ def __init__(self, **kwargs) -> None:
4241
self.client = docker.DockerClient(base_url=docker_host)
4342
else:
4443
self.client = docker.from_env(**kwargs)
44+
self.client.api.headers["x-tc-sid"] = SESSION_ID
4545

4646
@ft.wraps(ContainerCollection.run)
4747
def run(
@@ -50,6 +50,7 @@ def run(
5050
command: Optional[Union[str, list[str]]] = None,
5151
environment: Optional[dict] = None,
5252
ports: Optional[dict] = None,
53+
labels: Optional[dict[str, str]] = None,
5354
detach: bool = False,
5455
stdout: bool = True,
5556
stderr: bool = False,
@@ -65,10 +66,9 @@ def run(
6566
detach=detach,
6667
environment=environment,
6768
ports=ports,
69+
labels=create_labels(image, labels),
6870
**kwargs,
6971
)
70-
if detach:
71-
atexit.register(_stop_container, container)
7272
return container
7373

7474
def port(self, container_id: str, port: int) -> int:
@@ -143,14 +143,5 @@ def read_tc_properties() -> dict[str, str]:
143143
tuples = []
144144
with open(file) as contents:
145145
tuples = [line.split("=") for line in contents.readlines() if "=" in line]
146-
settings = {**settings, **{item[0]: item[1] for item in tuples}}
146+
settings = {**settings, **{item[0].strip(): item[1].strip() for item in tuples}}
147147
return settings
148-
149-
150-
def _stop_container(container: Container) -> None:
151-
try:
152-
container.stop()
153-
except NotFound:
154-
pass
155-
except Exception as ex:
156-
LOGGER.warning("failed to shut down container %s with image %s: %s", container.id, container.image, ex)

core/testcontainers/core/labels.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
from typing import Optional
2+
from uuid import uuid4
3+
4+
from testcontainers.core.config import RYUK_IMAGE
5+
6+
SESSION_ID: str = str(uuid4())
7+
LABEL_SESSION_ID = "org.testcontainers.session-id"
8+
LABEL_LANG = "org.testcontainers.lang"
9+
10+
11+
def create_labels(image: str, labels: Optional[dict[str, str]]) -> dict[str, str]:
12+
if labels is None:
13+
labels = {}
14+
labels[LABEL_LANG] = "python"
15+
16+
if image == RYUK_IMAGE:
17+
return labels
18+
19+
labels[LABEL_SESSION_ID] = SESSION_ID
20+
return labels

core/tests/test_ryuk.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
from testcontainers.core import container
2+
from testcontainers.core.container import Reaper
3+
from testcontainers.core.container import DockerContainer
4+
from testcontainers.core.waiting_utils import wait_for_logs
5+
6+
7+
def test_wait_for_reaper():
8+
container = DockerContainer("hello-world").start()
9+
wait_for_logs(container, "Hello from Docker!")
10+
11+
assert Reaper._socket is not None
12+
Reaper._socket.close()
13+
14+
assert Reaper._container is not None
15+
wait_for_logs(Reaper._container, r".* Removed \d .*", timeout=30)
16+
17+
Reaper.delete_instance()
18+
19+
20+
def test_container_without_ryuk(monkeypatch):
21+
monkeypatch.setattr(container, "RYUK_DISABLED", True)
22+
with DockerContainer("hello-world") as cont:
23+
wait_for_logs(cont, "Hello from Docker!")
24+
assert Reaper._instance is None

modules/arangodb/testcontainers/arangodb/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ class ArangoDbContainer(DbContainer):
2626
>>> from testcontainers.arangodb import ArangoDbContainer
2727
>>> from arango import ArangoClient
2828
29-
>>> with ArangoDbContainer("arangodb:3.9.1") as arango:
29+
>>> with ArangoDbContainer("arangodb:3.11.8") as arango:
3030
... client = ArangoClient(hosts=arango.get_connection_url())
3131
...
3232
... # Connect

0 commit comments

Comments
 (0)