PEP 668 support breaks --user/--editable

[stefanor]

Description

@obfusk filed Debian bug #1030335 about the implementation of PEP 668 support.

I think it's best if this is discussed here, with upstream involvement.
cc @geofft @doko42 @FFY00 @dstufft @uranusjr @ehashman @pradyunsg @kitterma who were involved in the PEP or pip in Debian.

Hi!

I just updated python3-pip and was greeted with a NEWS message about
PEP 668 support:

This version of pip introduces PEP 668 support. Debian's python3.11
interpreter will soon (>= 3.11.1-3) declare the installation to be
EXTERNALLY-MANAGED, instructing pip to disallow package installation outside
virtualenvs.

See: https://peps.python.org/pep-0668/

Practically, this means that you can't use pip to install packages outside a
virtualenv, on a Debian system, any more.

See /usr/share/doc/python3.11/README.venv for more details.
If that isn't available yet, check:
https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Not being able to install packages system-wide seems like a good idea,
I fully support that.

But I often install the Python packages I'm developing under my own
user account with "pip install -e", which is also made impossible by
this change.

And I intentionally do not use virtualenvs for this because I want to
have all dependencies provided by Debian packages, not downloaded from
PyPI.

And as far as I can tell there is not even an option I can give to pip
to tell it to allow me to install to ~/.local anyway. PEP 668
mentions a --break-system-packages (as an example), but such an option
doesn't seem to actually exist.

• FC

This bug is about a Debian-user-specific issue (not wanting to depend on PyPI), but I don't think this is something Debian should try to solve, alone.

When the PEP was drafted, we were assuming there'd be some sort of mechanism to override it (allow the user to break their system and keep both pieces), I quote:

The installer should have a way for the user to override these rules, such as a command-line flag --break-system-packages. This option should not be enabled by default and should carry some connotation that its use is risky.

That didn't make it into the implementation, so I suggested deleting the EXTERNALLY-MANAGED file in the documentation Debian is providing about this. It's not great, but it's about the only suggestion I can make:

https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Does pip have any thoughts on providing an override mechanism?

There is some value in not having an override (we don't need to keep testing that all these schemes work, for one). But as we transition, we're going to be breaking use-cases...

Expected behavior

No response

pip version

23.0

Python version

3.11.1

OS

Debian GNU/Linux

How to Reproduce

1. apt install python3-pip
2. check out a python source package.
3. pip install -e .

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[obfusk]

FWIW, I can use e.g. python3 -mvenv --system-site-packages --without-pip ~/.venv as a workaround (and even automatically activate that venv in ~/.bashrc).

But that's more work, and not compatible with what's already installed in ~/.local.

I don't see how that is any better than continuing to allow --user.

I'm still using the system site packages like before, so I don't think it "breaks the system" any less than --user does.

[obfusk]

I suggested deleting the EXTERNALLY-MANAGED file in the documentation Debian is providing about this. It's not great, but it's about the only suggestion I can make:

Which is only an option for admins, not regular users on e.g. a shared university server.

[uranusjr]

Disallowing --user is intentional because user-site package installations affect the system Python the same way system-site ones do (except they only affect the single user). Using a virtual environment with --system-site-packages is a much better solution.

[obfusk]

Disallowing --user is intentional

According to the PEP, so is having an override mechanism. Which there isn't.
So the current unconditional breakage of --user is not in accordance with the PEP.

Using a virtual environment with --system-site-packages is a much better solution.

Why?
I can understand requiring some kind of "I know what I'm doing" option (as the PEP mentions), after all --system-site-packages is also opt-in.
But why is is better to be forced to use a virtual environment when I always want that to be active?

[uranusjr]

Since the override mechanism is backed by the PEP, contribution is always welcomed.

[actualben]

If you're breaking PEP 370 shouldn't its status be marked as Superseded as part of this change?
Sorry I've just realized that this isn't quite the right place for this question.

[Omarabdul3ziz]

Thank you for the quick report and fix. but is the fix published yet? I tried to use the flag --break-system-packages but got error no such option

I am working on alpine:edge container

[uranusjr]

@actualben Quick answer is no. Marking PEP 370 as Superceded would be mean the mechanism no longer works, but what PEP 668 specifies is a way to disable user-site by default. Note that per-user site-packages can already be disabled via various ways, the new PEP only adds one additional way and encourages distros to set it by default.

[pradyunsg]

Thank you for the quick report and fix. but is the fix published yet?

No, it is not.

[futzu]

Deleting /usr/lib/pypy3.9/EXTERNALLY-MANAGED
fixes my problems with pip and pypy3 on Debian.

[pfmoore]

Note that while "practicality beats purity" and all that, and we don't yet have the override flag released, I feel it's important to note that deleting EXTERNALLY-MANAGED is not the intended way of overriding this protection, and should be treated as a temporary hack to work around the current lack of an override flag.

If you want to install into the system site-packages (and understand the risks¹), I would recommend that once the next version of pip is released with the override flag, you either use the --break-system-packages flag while invoking pip, or you set the PIP_BREAK_SYSTEM_PACKAGES environment variable just for the duration of your build (don't leave it set in the resulting runtime container!).

Footnotes

1. Of course, we can't dictate how people choose to build their containers, and we do understand that the container environment is atypical in many ways, but please understand the risks before overriding things. ↩

[obfusk]

I do not want to install into system site-packages, but I do want to continue to keep using --user (since a venv doesn't work for my use case), so I added this to my ~/.config/pip/pip.conf:

[global]
break-system-packages = true

[cr1ogen]

Thank you for the quick report and fix. but is the fix published yet? I tried to use the flag --break-system-packages but got error no such option

I am working on alpine:edge container

The command is pip install tmuxp --break-system-packages

[pradyunsg]

No, both of those invocations work. This flag was added in a newer version of pip than what the user was trying with.

[cr1ogen]

Only the second option works for me pip install tmuxp --break-system-packages in debian sid. I will wait for a new update to testing it

[TheGentleChainsaw]

@obfusk For what it's worth, I've decided to get around this by just setting up a venv in ~/.local, which you can absolutely do if you already have a ~/.local full of userland Python packages since it'll automatically convert them to execute inside the venv even if you run them from the normal shell. Also, because we're not looking for full isolation, we can safely use the system-site-packages flag when initialising the venv to avoid unnecessarily duplicating stuff from the system install.

IMHO, the perfect way to take advantage of a venv in set-and-forget mode where you don't have to use the activator script everytime you want to run a Python program. (Which reminds me, I should ask the virtualenv people about maybe adding an --activators none option...)

[actualben]

FWIW there is a discussion thread for this PEP: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302 -- it contains a lot of information about the history of this change and discussion of the options. There is also good engagement from the PEP authors / core python devs.

[pradyunsg]

Thanks @actualben for that link.

Locking this since there isn’t much more worth discussing on this issue tracker, and vitriol directed at the volunteers (who maintain the foundational pieces of software you rely on!) isn’t welcome.