Release builds fail with PyO3 auto-initialize due to static Python linking

[joshuadavidthomas]

Ah damn, I thought I had figured it out. The recent changes in #118 and #119 work in the test environment, but not in the release environment.

Relevant error logs:

  error: The `auto-initialize` feature is enabled, but your python installation only supports embedding the Python interpreter statically. If you are attempting to run tests, or a binary which is okay to link dynamically, install a Python distribution which ships with the Python shared library.

  Embedding the Python interpreter statically does not yet have first-class support in PyO3. If you are sure you intend to do this, disable the `auto-initialize` feature.

  For more information, see https://pyo3.rs/v0.24.2/building-and-distribution.html#embedding-python-in-rust

[joshuadavidthomas]

Okay, threw this to the 🤖 robot (Gemini 2.5 Pro specifically).. May help? May have hallucinated? Who knows, but this could at least be a starting point for investigating what's going on. (Maybe I'll learn a thing or two about dynamic vs static Python linking and embedding.. maybe not. 🤣)

---

This error message is definitely pointing towards a mismatch between how PyO3 expects to link Python (dynamically, especially with auto-initialize) and the specific Python distributions available in some of the CI environments used for building the release wheels.

Here's a breakdown of the situation and some potential paths forward:

1. The Core Problem: Static vs. Dynamic Python Libraries

   • The auto-initialize feature in PyO3 simplifies things by managing Py_Initialize for us, but it relies on being able to dynamically load the Python interpreter using its shared library (libpythonX.Y.so, .dylib, or .dll).
   • Some Python distributions, particularly those optimized for size or specific environments (like potentially some found in manylinux/musllinux build images or certain minimal installs), might only ship with the static library (libpythonX.Y.a). PyO3 doesn't have first-class support for embedding via static linking and warns against it.
   • The error occurs during the maturin-action step because Maturin is trying to build the wheel against one of these Python installations that lacks the required shared library.

2. Idea 1: Ensure Python Shared Library in CI (Preferred)

   • The most robust solution is likely to ensure that the Python environment used by Maturin always includes the shared library.
   • Investigation: We need to check which specific targets in the release.yml matrix are failing. Is it primarily the manylinux or musllinux builds? Or specific architectures?
   • Possible Fixes:
     • actions/setup-python: Ensure we're requesting standard CPython builds. The default usually works, but maybe specific versions/runners behave differently. Using python-implementation: CPython might help guarantee standard builds.
     • maturin-action / Build Images: For manylinux/musllinux builds, maturin-action uses specific Docker images (e.g., quay.io/pypa/manylinux...). It's possible the default Python installations in some of these images lack the shared library. We might need to:
       • Check if maturin-action has options to influence the Python build used within the container.
       • Potentially use a different manylinux image tag known to include shared libraries (e.g., non-slim variants if available).
       • See if PyO3's build configuration (pyo3-build-config) can be influenced via environment variables during the Maturin build to select a specific interpreter or provide hints. The PYO3_CONFIG_FILE or PYO3_PYTHON env vars might be relevant within the build container.
     • Python Build Flags: Sometimes Python is compiled with --enable-shared. We need to ensure the Pythons used by Maturin have this.

3. Idea 2: Disable auto-initialize (Fallback)

   • As the error message suggests, disabling the auto-initialize feature in crates/djls-project/Cargo.toml would bypass the immediate error.
   • Downside: This significantly increases complexity. We would need to manually call Py_Initialize before any Python interaction and potentially Py_Finalize on exit. This is tricky because:
     • The code runs both as a Python module (where Python is already initialized) via the djls entrypoint and as a Rust binary (the LSP server) that might need to initialize Python itself.
     • Managing the initialization state correctly across tests, the main binary, and the Python module entrypoint would be fragile.
   • Given these complexities, this should probably be avoided unless Idea 1 proves impossible.

Recommendation:

Let's focus on Idea 1. We should investigate the specific failing targets in the release.yml workflow and determine why the Python environment lacks the shared library in those cases. The solution likely involves configuring actions/setup-python or the environment within the maturin-action (especially for manylinux/musllinux) to ensure a Python distribution with the shared library is consistently used. Checking the PyO3/maturin-action documentation and potentially the PyPA manylinux image details might provide clues.

[joshuadavidthomas]

Yeah, so clearly auto-initialize is the core issue here. If I look across other Rust/Python libraries, specifically the ones linked in the maturin-action README and django-rusty-templates, it doesn't appear that any of them use that feature.

PyO3 specifically calls out new users using it when testing (ok, RUDE 😄):

The auto-initialize feature is deliberately disabled when embedding the interpreter statically because this is often unintentionally done by new users to PyO3 running test programs. Trying out PyO3 is much easier using dynamic embedding.
https://pyo3.rs/v0.24.2/building-and-distribution.html#statically-embedding-the-python-interpreter

django-rusty-templates has a clue in its test suite, by using pyo3::prepare_freethreaded_python() before calling Python::with_gil. Example -> https://github.com/LilyFoote/django-rusty-templates/blob/1c1cdec16e9cd2b9dfda3031baf77a862f05ba06/src/parse.rs#L1058

I'm going to give that a try.

[joshuadavidthomas]

OH YEAH 🎉

That seemed to do the trick. Had to do some switching around in the new build script in the new djls-dev crate to conditionally link whether or not the package was being build as an extension module (thanks Gemini again), but I think I've got this working now. auto-initialize has been removed and we're using pyo3::prepare_freethreaded_python() everywhere we need.