pip imports the deprecated distutils module

[vstinner]

Description

The pip/_internal/locations/_distutils.py file imports the distutils module:

pip/src/pip/_internal/locations/_distutils.py

Lines 9 to 12 in cb24fb4

	from distutils.cmd import Command as DistutilsCommand
	from distutils.command.install import SCHEME_KEYS
	from distutils.command.install import install as distutils_install_command
	from distutils.sysconfig import get_python_lib

This module is deprecated since Python 3.10 and is going to be removed in Python 3.12: python/cpython#92584 (comment)

Expected behavior

No response

pip version

22.0.4

Python version

3.12.0a0

OS

Linux

How to Reproduce

1. Build Python 3.12 with [WIP] gh-92584: Rename the distutils package to _distutils python/cpython#92585 PR
2. Create a venv without pip: ./python -m venv env
3. Install pip: ./env/bin/python -m ensurepip -v

Output

vstinner@apu$ ./env/bin/python -m ensurepip -v
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/home/vstinner/python/main/Lib/runpy.py", line 226, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 98, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/__main__.py", line 29, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/main.py", line 9, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 10, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/main_parser.py", line 8, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 23, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/parser.py", line 12, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/configuration.py", line 26, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/utils/logging.py", line 27, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/utils/misc.py", line 39, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/locations/__init__.py", line 14, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/locations/_distutils.py", line 9, in <module>
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 276, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 192, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 92, in _run_pip
    return subprocess.run([sys.executable, '-W', 'ignore::DeprecationWarning',
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/subprocess.py", line 558, in run
    raise CalledProcessError(retcode, process.args,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subprocess.CalledProcessError: Command '['/home/vstinner/python/main/env/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmps2pazori/setuptools-58.1.0-py3-none-any.whl\', \'/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmps2pazori\', \'-v\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

This module is provided for backward compatibility; on 3.10 it should never be used. So I guess (assuming there's no real bug where we still use it) we simply need to do conditional imports better.

[vstinner]

Maybe only import locations/_distutils.py in locations/__init__.py if _USE_SYSCONFIG is false?

[uranusjr]

I looked at the code and pip in fact still relies on some distutils internals beside the locations module. I’ll try to either remove them, or at least localise so distutils is imported only when relevant functionalities are accessed (the most significant usage of it is to parse --install-option).

[vstinner]

Hi, is there an update on this issue? How can I help?

[pradyunsg]

No updates (since those would be reflected on this issue).

Looking at this now... There's 5 major files that this needs changes for:

• src/pip/_internal/locations/init.py: this needs a conditional import of pip._internal.locations._distutils
• src/pip/_internal/locations/_distutils.py: well... this should be imported conditionally and never imported on 3.12+.
• src/pip/_internal/locations/_sysconfig.py: this needs change_root from distutils.util
• src/pip/_internal/operations/install/legacy.py: this needs change_root from distutils.util
• src/pip/_internal/utils/distutils_args.py: this uses FancyGetopt from distutils to parse out distutils arguments.

Help would be appreciated with all of these. There's one import/use of distutils in a vendored package -- for distilb.get_archive_formats -- and that code path shouldn't be executed in the real world (that function is not used by pip, AFAIK).

[pradyunsg]

I'll pick up the change_root changes, partly because that's the easiest of the lot. :)

[pradyunsg]

Alrighty, other than the open PRs, the only thing remaining here is moving to a conditional import of the _distutils module, in the locations package. If someone else could pick that up, that'd be great! ^>^

[auvipy]

distutils.sysconfig import get_python_lib, what should be new import for get_python_lib or what else should be used?

[uranusjr]

sysconfig.get_path("purelib") and sysconfig.get_path("platlib").

[q0w]

and what for get_prefixed_libs?

[uranusjr]

sysconfig does not provide a native way to supply a custom prefix, but you can use the vars argument and supply custom base, platbase, etc. values. (The semantic of “prefix” in distutils is split into quite several variables in sysconfig; you need to hunt them all up from the sysconfig source.)

https://github.com/python/cpython/blob/main/Lib/sysconfig.py

[pradyunsg]

This still needs a PR adding a conditional import to pip._internal.locations, to not import distutils if sysconfig is going to be used.

[q0w]

like this?

if _USE_SYSCONFIG:
    from . import _distutils

[pradyunsg]

It'd likely need more changes, but yes -- we basically want to avoid importing distutils if _USE_SYSCONFIG is true.

We should also have corresponding tests, to catch us regressing on this FWIW.

[ssbarnea]

Apparently the problem still reproduces with py311 from deadsnakes on ubuntu while trying to install a package.

cmd: /media/psf/c/a/ansible-navigator/.tox/u1/py311/bin/python -m pip install '.[test]'
Traceback (most recent call last):
...
  File "/media/psf/c/a/ansible-navigator/.tox/u1/py311/lib/python3.11/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/media/psf/c/a/ansible-navigator/.tox/u1/py311/lib/python3.11/site-packages/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils.cmd'

Does anyone know a workaround for it as I would really want to enable CI/CD pipeline for py311. I already had success on other projects but on this project, it even fails to install dependencies.

[sbidoul]

Unless someone plans to work on this in the coming days, I'm going to push this to down to 22.3.

[uranusjr]

I’ll try to find some time this weekend.

[pradyunsg]

@uranusjr Hmm... does this need anything more than the following?

diff --git a/src/pip/_internal/locations/__init__.py b/src/pip/_internal/locations/__init__.py
index 99312d77a..22e90a00f 100644
--- a/src/pip/_internal/locations/__init__.py
+++ b/src/pip/_internal/locations/__init__.py
@@ -11,7 +11,7 @@
 from pip._internal.utils.deprecation import deprecated
 from pip._internal.utils.virtualenv import running_under_virtualenv
 
-from . import _distutils, _sysconfig
+from . import _sysconfig
 from .base import (
     USER_CACHE_DIR,
     get_major_minor_version,
@@ -241,6 +241,8 @@ def get_scheme(
     if _USE_SYSCONFIG:
         return new
 
+    from . import _distutils
+
     old = _distutils.get_scheme(
         dist_name,
         user=user,

[uranusjr]

Maybe not? I am not sure.

[uranusjr]

Actully there is one more module we need to fix.