regression in 1.4.0-rc1

[baxelrod-bdai]

🐞 bug report

Affected Rule

bazel build //... errors out when using rules_python 1.4.0-rc1 (and rc0) on a small repository.

Is this a regression?

Yes, rules_python 1.3.0 builds fine.

Description

The build error:

baxelrod@baxelrod-ThinkPad:~/repos/rai-bazel-sandbox$ bazel build //...
ERROR: no such package '@@rules_python++pip+pip//importlib_metadata': BUILD file not found in directory 'importlib_metadata' of external repository @@rules_python++pip+pip. Add a BUILD file to a directory to mark it as a package.
ERROR: /home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python++pip+pip_310_build/BUILD.bazel:6:34: no such package '@@rules_python++pip+pip//importlib_metadata': BUILD file not found in directory 'importlib_metadata' of external repository @@rules_python++pip+pip. Add a BUILD file to a directory to mark it as a package. and referenced by '@@rules_python++pip+pip_310_build//:whl'
ERROR: Analysis of target '//:modules_map_pip' failed; build aborted: Analysis failed
INFO: Elapsed time: 15.117s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
INFO: Build Event Protocol files produced successfully.
Loading: 300 packages loaded
    Fetching repository @@rules_python++pip+pip_u2204_310_argparse; starting 14s
    Fetching repository @@rules_python++pip+pip_u2204_310_numpy; starting 14s
    Fetching repository @@rules_python++pip+pip_310_minilog; starting 14s
    Fetching repository @@rules_python++pip+pip_u2204_310_spatialmath_python; starting 14s
    Fetching ... @@rules_python++pip+pip_310_jinja2; Running whl_library.ResolveRequirement(rules_python++pip+pip_310_jinja2, jinja2==3.1.6) 14s
    Fetching repository @@rules_python++pip+pip_310_numpy; starting 14s
    Fetching repository @@rules_python++pip+pip_310_markupsafe; starting 14s
    Fetching repository @@rules_python++pip+pip_310_google_cloud_bigquery; starting 14s ... (116 fetches)

I'm running rules_uv like this:

pip_compile(
    name = "generate_requirements_latest",
    extra_args = [
        "--prerelease=allow",
        "--quiet",
        "--universal",
        "--upgrade",
    ],
    requirements_in = "requirements_latest.in",
    requirements_txt = "requirements_latest.lock",
)

Relevant versions:

bazel_dep(name = "aspect_rules_py", version = "1.3.2")
bazel_dep(name = "rules_python", version = "1.4.0-rc1") #1.3.0
bazel_dep(name = "rules_uv", version = "0.68.0")

On Linux Ubuntu 22.04, x86.

🔬 Minimal Reproduction

I'm not sure how to minimally reproduce. But my requirements.in file has only:

argparse >= 1.4.0
certifi >= 2025.1.31
charset-normalizer >= 3.4.1
datadog_api_client >= 2.23.0
google-auth >= 2.38.0
google-cloud-bigquery >= 3.31.0
google-cloud-storage >= 3.1.0
idna >= 3.10
matplotlib >= 3.10.1
minilog >= 2.3.1
numpy >= 1.22    ; platform_machine == 'aarch64'
numpy >= 2.2.4   ; platform_machine == 'x86_64'
pip-tools >= 7.4.1
pyqt5 >= 0.0     ; platform_machine == 'aarch64'
pyqt5 >= 5.15.11 ; platform_machine == 'x86_64'
pytest >= 8.3.5
requests >= 2.32.3
spatialmath-python >= 1.1.14 ; platform_system == 'Linux'
toml >= 0.10.2
torch >= 2.3
urllib3 >= 2.3.0

[rickeylev]

This sounds like the bug @keith and @aignas just fixed with #2806

Judging the the error, it sounds like the build package is trying to reference the importlib_metadata package.

I can't tell which version of build is being used, but here's the latest METADATA file: https://pypi-browser.org/package/build/build-1.2.2.post1-py3-none-any.whl/build-1.2.2.post1.dist-info/METADATA

[rickeylev]

Here's an easier repro:

git clone https://github.com/rickeylev/rules_python/tree/repro.2815
cd examples/scratch
bazel build //...

Should result in an error similar to reported. I saw errors relating to both toml/tomli missing and importlib_metadata.

[rickeylev]

Hm, I'm not sure if this is a bug with rules_uv or some idiosyncracy of pip that was covering something up?

requirements.txt, generated by rules_uv, was missing importlib_metadata. So it's not surprising that no BUILD file gets generated for it, and thus @pip//importlib_metadata doesn't exist.

At the same time, build has a dependency on it, sort of. Maybe? From the requires_dist list in the generated build file:

        "importlib-metadata >= 4.6; python_full_version < \"3.10.2\"",
        "importlib-metadata >= 5.1 ; extra == \"typing\"",

Guess as to what's happening:

• rules_uv is running Python greater than 3.10.2. Thus it ignores that line
• The extra==typing isn't activated, so it ignores that too
• And thus, requirements.txt doesn't contain importlib_metadata
• The pip.parse() logic thinks the python version is less than 3.10.2, so it includes the line, and thus adds the importlib_metadata dependency to deps. This can be seen with bazel query --output=build @@rules_python++pip+pip_310_build//:pkg, and carefully looking for the deps= part, which shows @pip//importlib_metadata:pkg as a dependency.

The actual python version used, in this build anyways, is 3.10.16.

---

A workaround might be to add importlib_metadata to requirements.in. This will get the repo generated, however, it also means importlib_metadata will be part of the dependencies.

---

For reference, excerpt foom external/rules_python++pip+pip_310_build/BUILD.bazel
(note: i sorted this list, it's not in this order in the build file)

requires_dist = [
        "build[uv, virtualenv] ; extra == \"test\"",
        "build[uv] ; extra == \"typing\"",
        "colorama; os_name == \"nt\"",
        "filelock >= 3 ; extra == \"test\"",
        "furo >= 2023.08.17 ; extra == \"docs\"",
        "importlib-metadata >= 4.6; python_full_version < \"3.10.2\"",
        "importlib-metadata >= 5.1 ; extra == \"typing\"",
        "mypy ~= 1.9.0 ; extra == \"typing\"",
        "packaging >= 19.1",
        "pyproject_hooks",
        "pytest >= 6.2.4 ; extra == \"test\"",
        "pytest-cov >= 2.12 ; extra == \"test\"",
        "pytest-mock >= 2 ; extra == \"test\"",
        "pytest-rerunfailures >= 9.1 ; extra == \"test\"",
        "pytest-xdist >= 1.34 ; extra == \"test\"",
        "setuptools >= 42.0.0 ; extra == \"test\" and ( python_version < \"3.10\")",
        "setuptools >= 56.0.0 ; extra == \"test\" and ( python_version == \"3.10\")",
        "setuptools >= 56.0.0 ; extra == \"test\" and ( python_version == \"3.11\")",
        "setuptools >= 67.8.0 ; extra == \"test\" and ( python_version >= \"3.12\")",
        "sphinx ~= 7.0 ; extra == \"docs\"",
        "sphinx-argparse-cli >= 1.5 ; extra == \"docs\"",
        "sphinx-autodoc-typehints >= 1.10 ; extra == \"docs\"",
        "sphinx-issues >= 3.0.0 ; extra == \"docs\"",
        "tomli ; extra == \"typing\"",
        "tomli >= 1.1.0; python_version < \"3.11\"",
        "typing-extensions >= 3.7.4.3 ; extra == \"typing\"",
        "uv >= 0.1.18 ; extra == \"uv\"",
        "virtualenv >= 20.0.35 ; extra == \"virtualenv\"",
        "wheel >= 0.36.0 ; extra == \"test\"",
]

[aignas]

Ah this is related to #2319.

So the short story is:

• The default python version is 3.10.16, but because of [pypi] Support python micro version in env marker based dependency evaluation #2319, the version we use in parsing the METADATA is 3.10.0. That is why the importlib-metadata is mistakenly added to the dependency closure - it should not be there.
• In fix: use the python micro version to parse whl metadata in bzlmod #2793 I am fixing this to correctly pass the full python version around. I think I've just fixed one of the remaining things to make the CI green (it is still running), so the PR might be ready for testing to see if it fixes the regression mentioned in this issue.

If the CI in the PR is green, could you please (@baxelrod-bdai) check if adding it as a patch fixes the issue you are seeing?

[aignas]

EDIT: Opened #2820 for cherry-picking the changes.

[baxelrod-bdai]

I tested with the patch and it seems to work. Thanks!