Python 3.10: pip install -r requirements.txt fails due to unpinned importlib_resources

[salticus]

This is a small issue but I thought the sigstore team might like to be aware of it.

Summary

On Python 3.10, installing with requirements.txt fails, reporting that importlib_resources is unpinned (== vs ~=).

pip install -r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt 
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
   importlib_resources~=5.7 from https://files.pythonhosted.org/packages/38/71/c13ea695a4393639830bf96baea956538ba7a9d06fcce7cef10bfff20f72/importlib_resources-5.12.0-py3-none-any.whl#sha256=7b1deeebbf351c7578e09bf2f63fa2ce8b5ffec296e0d349139d43cca061a81a (from sigstore==1.1.0->-r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt (line 363))

On Python 3.11 installing with sigstore's requirements.txt succeeds.

Since the Python Sigstore Information page suggests that this command will provide "install-time assurances including hash-checking", and sigstore itself is meant to secure the software supply chain, the error is mildly alarming.

Permalink to today's requirements.txt: https://github.com/sigstore/sigstore-python/blob/27a442a8b24aa63a6613c1296d52f0d905740f52/install/requirements.txt

Error Case

python3 -m venv venv
source ./venv/bin/activate

python -VV
# Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]

python -m pip --version
# pip 22.0.2 from [path removed]/venv/lib/python3.10/site-packages/pip (python 3.10)

python -m pip install -r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt
# ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#    importlib_resources~=5.7 from https://files.pythonhosted.org/packages/38/71/c13ea695a4393639830bf96baea956538ba7a9d06fcce7cef10bfff20f72/importlib_resources-5.12.0-py3-none-any.whl#sha256=7b1deeebbf351c7578e09bf2f63fa2ce8b5ffec296e0d349139d43cca061a81a (from sigstore==1.1.0->-r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt (line 363))

cat /etc/os-release | grep PRETTY_NAME
# PRETTY_NAME="Ubuntu 22.04.2 LTS"

Possibly relevant project information

• pyproject.toml importlib_resources line contains the line

  importlib_resources ~= 5.7; python_version < '3.11'

• Pull 341 was to 'eliminate deprecation warnings on 3.11... [it] uses importlib_resources as a "polyfill" on older Python versions [permitting use of] the new resources.files() APIs. (These uses will be mostly removed with the TUF work in Retrieve CTFE signing key via TUF #25...'

Other observations

The following might end up as issues on the pypa/pip project, but are included in case they are relevant.

A. The error only appears in a fresh environment.

pip install sigstore
# Collecting sigstore
# [content removed for brevity]

python -m pip install --require-hashes -r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt
# [content removed for brevity]
# Requirement already satisfied: importlib_resources~=5.7 in ./venv/lib/python3.10/site-packages (from sigstore==1.1.0->-r https://raw.githubusercontent.com/sigstore/sigstore-python/main/install/requirements.txt (line 363)) (5.12.0)
# [No error, despite the --require-hashes argument]
# Note the `~=` which apparently contradicts the error's message that "In --require-hashes mode, all requirements must have their versions pinned with =="

B. --require-hashes is not part of the command

The pip install command producing the error does not use '--require-hashes' in the first place. (The '--require-hashes' mode is probably activated by the hash format of the requirements file.)

C. The requirements.txt does not mention importlib_resources

The requirements.txt does not mention importlib_resources. The pyproject.toml does mention importlib_resources as a conditional import for python versions under 3.11. Running the same command in 3.11 does not produce an error.

1. But none of the dependency versions in pyproject.toml are pinned so why is importlib_resources special?
2. Why is pip trying to install a dependency which is not listed in requirements.txt in the first place?

D. Over time importlib_resources becomes part of CPython.

The importlib_resources library is a testing ground for later inclusion in CPython. The wheel mentioned in the error message contains the following in its file, importlib_resources-5.12.0.dist-info/METADATA

Compatibility
=============

New features are introduced in this third-party library and later merged
into CPython. The following table indicates which versions of this library
were contributed to different versions in the standard library:

.. list-table::
   :header-rows: 1

   * - importlib_resources
     - stdlib
   * - 5.9
     - 3.12
   * - 5.7
     - 3.11
   * - 5.0
     - 3.10
   * - 1.3
     - 3.9
   * - 0.5 (?)
     - 3.7

[di]

Thanks for the issue. We just forgot to recompile the file in #341 and didn't have any testing to ensure this is installable across all Python versions. #521 will resolve this.