Duplicate, conflicting imports due to changes with editable installs (Migrating from setup.py to pyproject.toml may subtly require enabling namespace support)

[ctheune]

• a detailed description of the bug or problem you are having

We've converted an older setuptools/setup.py based project to uv and pyproject.toml. The editable installation (using uv sync) uses this requirements.txt snippet:

-e /srv/s-dev/fc.directory[test]
    # via -r requirements.in
-e /srv/s-dev/fc.support
    # via
    #   -r requirements.in

This ends up with the following sys.path when starting the venv's interpreter:

['',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python312.zip',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python3.12',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python3.12/lib-dynload',
 '/srv/s-dev/deployment/work/directorybuild/.venv/lib/python3.12/site-packages',
 '__editable__.fc_directory-2.69.1.dev0.finder.__path_hook__',
 '/srv/s-dev/fc.support/src']

Now, in the actual application this works fine. Howevever, our tests fail because modules are imported twice and I can see that within the pytest environment the path for the namespace package appears a second time
The first dependency is added using a finder hook (something setuptools introduced a while ago it appears).

The second interestingly uses a plain .pth file and ends up working fine.

Here's the path from the pytest session:

(Pdb) pp sys.path
['/srv/s-dev/fc.directory/src/fc',
 '/srv/s-dev/deployment/work/directorybuild/.venv/bin',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python312.zip',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python3.12',
 '/nix/store/80v02bzfz0g507q8b1g9ylmygc572hzd-python3-3.12.7/lib/python3.12/lib-dynload',
 '/srv/s-dev/deployment/work/directorybuild/.venv/lib/python3.12/site-packages',
 '__editable__.fc_directory-2.69.1.dev0.finder.__path_hook__',
 '/srv/s-dev/fc.support/src',
 '/srv/s-dev/deployment/work/directorybuild/.venv/lib/python3.12/site-packages/setuptools/_vendor']

• output of pip list from the virtual environment you are using

Package               Version      Editable project location
--------------------- ------------ -------------------------
alembic               1.13.2
annotated-types       0.6.0
anyio                 4.4.0
apipkg                3.0.2
arrow                 1.2.3
attrs                 22.2.0
beaker                1.13.0
beautifulsoup4        4.12.3
black                 24.8.0
certifi               2024.7.4
cffi                  1.17.1
chameleon             4.5.4
chardet               4.0.0
charset-normalizer    2.0.12
click                 8.1.8
colander              2.0
colorama              0.4.6
consulate-fc-nix-test 1.1.0a1
coverage              7.2.7
cryptography          43.0.3
cssselect2            0.7.0
dataproperty          0.55.1
decorator             4.4.2
deform                2.0.8
defusedxml            0.7.1
deprecation           2.1.0
ecdsa                 0.19.1
exceptiongroup        1.1.2
execnet               1.7.1
fanstatic             1.5
fastapi               0.115.11
fc-directory          2.69.1.dev0  /srv/s-dev/fc.directory
fc-support            1.0.dev0     /srv/s-dev/fc.support
future                1.0.0
gocept-cache          6.0
gocept-collmex        2.1.0
gocept-testdb         6.0
gocept-testing        4.0
greenlet              3.0.3
grokcore-component    4.1.1
h11                   0.14.0
httpcore              0.16.3
httpx                 0.23.3
humanfriendly         4.18
hupper                1.12.1
icalendar             4.0.4
idna                  3.7
iniconfig             2.0.0
iso8601               0.1.16
jinja2                3.1.6
jwcrypto              1.5.6
lxml                  5.3.1
mako                  1.2.4
markdown              3.4.1
markdown2             2.5.3
markupsafe            2.1.2
martian               1.3.post1
mbstrdecoder          1.1.2
mock                  3.0.5
mypy-extensions       1.0.0
nagiosplugin          1.3.3
netaddr               0.7.19
oic                   1.6.1
outcome               1.2.0
packaging             23.0
passlib               1.7.4
paste                 3.10.1
pastedeploy           3.1.0
pathspec              0.9.0
pathvalidate          2.5.2
peppercorn            0.6
perfmetrics           3.3.0
pillow                10.4.0
plaster               1.1.2
plaster-pastedeploy   0.7
platformdirs          4.2.2
plone-testing         7.0.3
pluggy                1.5.0
pmw-py3               2.1
preppy                4.2.2
psycopg2              2.9.10
pyasn1                0.4.8
pyasn1-modules        0.2.7
pycparser             2.19
pycryptodomex         3.20.0
pydantic              2.6.3
pydantic-core         2.16.3
pydantic-settings     2.2.1
pygments              2.16.1
pyjwkest              1.4.2
pyotp                 2.3.0
pypdf                 4.3.1
pyramid               1.10.8
pyramid-beaker        0.8
pyramid-chameleon     0.3
pyramid-debugtoolbar  4.5.1
pyramid-deform        0.2
pyramid-fanstatic     0.6.dev0+fc0
pyramid-mailer        0.15.1
pyramid-mako          1.1.0
pyramid-retry         2.1
pyramid-rpc           0.8
pyramid-tm            2.6
pysocks               1.7.1
pyst2                 0.5.1
pytablewriter         0.64.2
pytest                8.3.2
pytest-cache          1.0
pytest-cov            4.1.0
pytest-localserver    0.8.0
pytest-rerunfailures  12.0
pytest-timeout        2.1.0
pytest-xdist          3.3.1
python-dateutil       2.8.2
python-dotenv         1.0.1
python-keycloak       3.12.0
python-ldap           3.4.3
pytz                  2024.2
pyyaml                6.0.2
redis                 4.5.5
reportlab             4.2.2
repoze-lru            0.7
repoze-sendmail       4.4.1
requests              2.32.3
requests-toolbelt     1.0.0
rfc3986               1.5.0
rlextra               4.2.2
selenium              4.10.0
setuptools            74.0.0
shortuuid             0.5.0
simplejson            3.18.3
six                   1.16.0
slowlog               0.9
sniffio               1.3.0
sortedcontainers      2.4.0
soupsieve             1.9.6
sqlalchemy            2.0.35
sshpubkeys            3.1.0
starlette             0.46.1
structlog             20.1.0
svglib                1.4.1
systemd-python        235
tabledata             1.3.1
tabulate              0.9.0
tcolorpy              0.1.2
tinycss2              1.2.1
transaction           3.0.1
translationstring     1.3
trio                  0.22.2
trio-websocket        0.10.3
typepy                1.3.0
typing-extensions     4.10.0
urllib3               1.26.19
validate-email        1.3
venusian              3.1.0
waitress              2.1.2
webencodings          0.5.1
webob                 1.8.8
webtest               3.0.0
werkzeug              3.0.3
wsgiproxy2            0.4.6
wsproto               1.2.0
xlsxwriter            1.2.6
zope-cachedescriptors 5.0
zope-component        6.0
zope-configuration    5.0.1
zope-deprecation      5.0
zope-dottedname       6.0
zope-event            5.0
zope-hookable         6.0
zope-i18nmessageid    6.1.0
zope-interface        7.0.3
zope-lifecycleevent   5.0
zope-location         5.0
zope-proxy            5.3
zope-schema           7.0.1
zope-security         7.1
zope-sendmail         6.2
zope-sqlalchemy       3.1
zope-testbrowser      7.0
zope-testing          5.0.1

• pytest and operating system versions

platform linux -- Python 3.12.7, pytest-8.3.2, pluggy-1.5.0

DISTRIB_CODENAME=uakari
DISTRIB_DESCRIPTION="NixOS 24.05 (Uakari)"
DISTRIB_ID=nixos
DISTRIB_RELEASE="24.05"
LSB_VERSION="24.05 (Uakari)"

• minimal example if possible

Sorry, no idea. I'm still trying to find a workaround for me ATM. Here's a number of issues I've been stumbling into and they seem to be pointing towards "client code needs to follow" as the answer ...

• uv pip install -e and its PEP-660-style editable installs break vscode import resolution (and probably static type checkers) astral-sh/uv#3898
• Due to limitation, static analysis don't support import hook used in editable install v64+ pypa/setuptools#3518
• Editable installs are broken after __editable__ and __path_hook__ changes pypa/setuptools#3548

[ctheune]

I'm trying to find a workaround and am noticing more details. Switching to importlib style importing didn't help.

But I noticed that this seems to be related to namespacing and relative imports. When importing something that was already imported from an absolute import, then I end up with a different module in the wrong namespace.

(Pdb) !from ... import utils as utils2
(Pdb) p utils2
<module 'directory.utils' from '/srv/s-dev/fc.directory/src/fc/directory/utils.py'>
(Pdb) p utils
<module 'fc.directory.utils' from '/srv/s-dev/fc.directory/src/fc/directory/utils.py'>

Taking a closer look at the sys.path (in prepend mode) we can see that the directory that is added to the path cuts off the namespace package "fc":

(Pdb) pp sys.path
['/srv/s-dev/fc.directory/src/fc',
 '/srv/s-dev/deployment/work/directorybuild/.venv/bin',
...
]

And we can see that in the test modules name:

(Pdb) u
> /srv/s-dev/fc.directory/src/fc/directory/customer.py(8)<module>()
(Pdb) p __name__
'directory.customer'

This causes relative imports to end up with different/wrong module names and thus create conflicts e.g. with sqlalchemy mapper construction in my case.

Aaaaand ... after going through the debugger it turns out I likely have to use consider_namespace_packages now (but didn't have to do that in the old-style setup.py-based structure) ...

So, I guess previously the way setuptools/uv/pip was setting up the namespaces was implicitly masking them.

The error mode is a bit weird so that if you have a working project and have never seen the above option then there isn't much to guide you through this.

[RonnyPfannschmidt]

i vaguely recall that old style namespaces use setuptools namespaces (that fix up import paths and add init.py files all over)

pytest still has a backlog wrt actual pep420 namespace support due to backward compatibility concerns that are hard to solve

[ctheune]

Yeah, I guess this issue isn't directly something to fix, maybe something to improve helping understand what's going on. It took me a good 2 hours figuring out what's going on, so this might also just be a bit of documentation for future victims ... ;)