Python 3.13.3 Circular import

[petrasvestartas]

I have circular import problem for relatively simple nanobind code.

Repository;
compas_shapeop

Installation

conda create -n compas_shapeop -c conda-forge python=3.9 compas -y
pip install -r requirements-dev.txt
pip install --no-build-isolation -ve . -Ceditable.rebuild=true
cibuildwheel --output-dir wheelhouse .

Circular import error happens during test phase.
Can anyone try this repository?

[LecrisUT]

cibuildwheel would be doing another build there, so are the other pip install relevant? The pip install numpy compas inside the tool.cibuildwheel.test-command is also probably undesired there since those are run after the package is installed.

[henryiii]

Could you paste the error?

[petrasvestartas]

We have even tried to do more simple direct python file import to remove all the possible issues, but then we got this error, that says that the c++ binding was never copied to the virtual environment, from 3.9 to 3.12 no error like this occurs:

ImportError while importing test module 'C:\brg\code\forks\compas_shapeop\tests\test_dynamicsolver.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\brg\code\forks\compas_shapeop\tests\test_dynamicsolver.py:3: in <module>
    from compas_shapeop.shapeop import Solver
..\venv-test\Lib\site-packages\compas_shapeop\shapeop.py:9: in <module>
    from ._shapeop import SolverWrapper
E   ModuleNotFoundError: No module named 'compas_shapeop._shapeop'

The place where it really does not work is here:
from ._shapeop import SolverWrapper

Can this be related of python itself?

python/cpython#123880

Locally python 3.9, 3.10, 3.11, 3.12, 3.13 also works, but not cibuildwheels.

[petrasvestartas]

@LecrisUT

Yes cibuildwheel --output-dir wheelhouse . would be enough.

But still python 3.13 error I cannot explain.

[LecrisUT]

Is editable install relevant? cibuildwheel should build with isolation. Can you check the contents of the generated wheel, and is it equivalent with pip wheel?

My first suspicion is that the install() don't point appropriately. (Edit: that part looks fine though, other than it will fail on windows)

[henryiii]

No, the "special" thing about 3.13 is you are using the Stable ABI for 3.12+, and so cibuildwheel is reusing the 3.12 wheel and just running the tests on it. This probably means you aren't making a valid stable ABI wheel.

[petrasvestartas]

@henryiii it can be true because on 3.13 build everything works, but it does not work when both 3.12 and 3.13 are built together.

If this is the case and Stabe ABI will support future python version.
How can I skip all the version above 3.12?

[henryiii]

Can you look at the filenames inside the wheel? (Might be in your logs). You can also remove the Stable ABI handling, that would quickly fix your issue, though you'd have to make separate wheels for 3.12 and 3.13 (and 3.14, etc).

[henryiii]

this is the case and Stabe ABI will support future python version

No, because this test is failing, the Stable ABI isn't working. I'm guessing it's not generating the correct name for the .so's, but this is nanobind_add_module which should work correctly.

[henryiii]

You can mimic cibuildwheel locally by building for 3.12, then installing the wheel on 3.13, by the way. That's all cibuildwheel is doing. (Unless it's the manylinux bundling that's broken, but it shouldn't be).

[petrasvestartas]

I commented out

function(add_nanobind_module module_name source_file)
  nanobind_add_module(
    ${module_name}
    # STABLE_ABI
    NB_STATIC
    ${source_file}
  )

I let the process run again, same problem on 3.13 version:

_________________________________________________________________________ ERROR collecting tests/test_dynamicsolver.py __________________________________________________________________________
ImportError while importing test module 'C:\brg\code\forks\compas_shapeop\tests\test_dynamicsolver.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\brg\code\forks\compas_shapeop\tests\test_dynamicsolver.py:3: in <module>
    from compas_shapeop.shapeop import Solver
..\venv-test\Lib\site-packages\compas_shapeop\shapeop.py:9: in <module>
    from ._shapeop import SolverWrapper
E   ModuleNotFoundError: No module named 'compas_shapeop._shapeop'

[LecrisUT]

The filenames inside the .whl is relevant here. If you comment out the STABLE_ABI you also need to comment out wheel.py-api. But it is weird that that combination does not work

[petrasvestartas]

This is the log, the 3.13 is skipped since 3.12 is compatible, but this is where the test fails: log.txt

[petrasvestartas]

@LecrisUT I've commented out the two, now the process is rerunning. Let's see. Fingers crossed.

[petrasvestartas]

Worked without STABLE_ABI:

[henryiii]

That's great, but the StableABI shouldn't be broken. Can you build a 3.12 with stable ABI (maybe you already have it), then look at the .so names with unzip -l <wheelname>? What are the .so's called? (You can do the same on a non-stable wheel if you want).

[petrasvestartas]

I added the screenshot above, I am currently on windows, the dynamic libraries of python ends by .pyd.

The abi wheel contains only 3.12 version:
_shapeop.cp312-win_amd64.pyd

This is repository (no need of any 3rd party installation as it is handled by cmake): https://github.com/compas-dev/compas_shapeop

[LecrisUT]

Ah, probably because you don't have Development.SABIModule component in the find_package(Python) ¹

Footnotes

1. https://github.com/compas-dev/compas_shapeop/blob/9f575799747d79348b2661917628408fa84acd09/CMakeLists.txt#L71 ↩

[petrasvestartas]

Just added, I am rerunning the build again :)

[petrasvestartas]

It works! Thank you a lot!