Support PyPy 3.7 officially and auto-test it in CI

PyPy might give 2-4x speedup on pure-Python scripts and save memory by default (as if slots were used in CPython). This enabled Kubernetes operators in CPU-/RAM-tight environments without special improvements and optimizations from Kopf's side. PyPy was unofficially supported and tested manually from time to time. With this change, let's make it official and automated.

MyPy is excluded from test-time dependencies: one of MyPy's sub-dependency ("typed-ast") fails at being installed (python/typed_ast#111). However, PyPy is not used for linting/type-checking, only CPython is. Therefore, MyPy is not needed for PyPy.

Signed-off-by: Sergey Vasilyev <[email protected]>

Author: nolar

.github/workflows/ci.yaml

@@ -38,7 +38,7 @@ jobs:
       fail-fast: false
       matrix:
         install-extras: [ "", "full-auth" ]
-        python-version: [ "3.7", "3.8", "3.9", "3.10" ]
+        python-version: [ "3.7", "3.8", "3.9", "3.10", "pypy-3.7" ]
     name: Python ${{ matrix.python-version }} ${{ matrix.install-extras }}
     runs-on: ubuntu-20.04
     timeout-minutes: 5  # usually 2-3 mins

.github/workflows/thorough.yaml

@@ -42,7 +42,7 @@ jobs:
       fail-fast: false
       matrix:
         install-extras: [ "", "full-auth" ]
-        python-version: [ "3.7", "3.8", "3.9", "3.10" ]
+        python-version: [ "3.7", "3.8", "3.9", "3.10", "pypy-3.7" ]
     name: Python ${{ matrix.python-version }} ${{ matrix.install-extras }}
     runs-on: ubuntu-20.04
     timeout-minutes: 5  # usually 2-3 mins

README.md

@@ -124,6 +124,10 @@ That easy! For more features, see the [documentation](https://kopf.readthedocs.i
 
 ## Usage
 
+Python 3.7+ is required:
+[CPython](https://www.python.org/) and [PyPy](https://www.pypy.org/)
+are officially supported and tested; other Python implementations can work too.
+
 We assume that when the operator is executed in the cluster, it must be packaged
 into a docker image with a CI/CD tool of your preference.
 

docs/install.rst

@@ -4,6 +4,10 @@ Installation
 
 .. highlight:: bash
 
+Prerequisites:
+
+* Python >= 3.7 (CPython and PyPy are officially tested and supported).
+
 To install Kopf::
 
     pip install kopf

kopf/_kits/webhooks.py

@@ -160,9 +160,11 @@ async def _serve_fn(request: aiohttp.web.Request) -> aiohttp.web.Response:
         runner = aiohttp.web.AppRunner(app, handle_signals=False)
         await runner.setup()
         try:
+            # Note: reuse_port is mostly (but not only) for fast-running tests with SSL sockets;
+            # multi-threaded sockets are not really used -- high load is not expected for webhooks.
             addr = self.addr or None  # None is aiohttp's "any interface"
             port = self.port or self._allocate_free_port()
-            site = aiohttp.web.TCPSite(runner, addr, port, ssl_context=context)
+            site = aiohttp.web.TCPSite(runner, addr, port, ssl_context=context, reuse_port=True)
             await site.start()
 
             # Log with the actual URL: normalised, with hostname/port set.

requirements.txt

@@ -14,7 +14,8 @@ freezegun
 import-linter
 isort>=5.5.0
 lxml
-mypy==0.910
+# Mypy requires typed-ast, which is broken on PyPy 3.7 (could work in PyPy 3.8).
+mypy==0.910; implementation_name == "cpython"
 pre-commit
 pyngrok
 pytest>=6.0.0

tests/reactor/test_queueing.py

@@ -16,6 +16,7 @@
 """
 import asyncio
 import contextlib
+import gc
 import weakref
 
 import async_timeout
@@ -224,6 +225,10 @@ async def test_garbage_collection_of_streams(settings, stream, events, unique, w
     # The jobs can take a tiny moment more, but this is noticeable in the tests.
     await asyncio.sleep(0.1)
 
+    # For PyPy: force the gc! (GC can be delayed in PyPy, unlike in CPython.)
+    # https://doc.pypy.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies
+    gc.collect()
+
     # Truly garbage-collected? Memory freed?
     assert all([ref() is None for ref in refs])