Bug on pip upgrading pip itself?

[lucbouge]

Dear pip_upgrade_outdated people,

I suspect that your script fails when pip itself is obsolete. In this case, pip3 is not updated correctly, and still requests the former version, which results in a strange failure: raise JSONDecodeError("Expecting value", s, err.value) from None.

I personally circumvent this problem by using pip_upgrade_outdated --exclude pip.

When pip is to be upgraded, I have to do it from outside: python3 -m pip install --upgrade pip. However, there may be of course better way of doing, for sure.

Please, could you at least let your pip_upgrade_outdated script exclude pip from being upgraded and issue a message?

Thank you for your excellent work,

Luc.

[defjaf]

Hi pip_upgrade_outdated user!

Thanks for using the app. Is this the same as this pip error in 19.2 which was fixed in the quickly updated 19.2.1? (If you're still on 19.2, try just doing pip list --outdated and you should see it without running pip_upgrade_outdated.)

[lucbouge]

Actually, I have seen this message, as I was tracking the bug exactly at the same time. But I think the two problems are different.

Please find below an extensive test trace. I attach a copy of the pip3 script after the pip install --upgrade pip command. As you can see, it has not been upgraded through the pip upgrade.

In some sense, the bug has nothing to do with pip_upgrade_outdated. What is needed from you is probably just not trying to update pip in the same way as you do for regular obsolete packages.

pip3.zip

Test trace

$ brew reinstall python

==> Reinstalling python 
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.4.mojave.bottle.
Already downloaded: /Users/bouge/Library/Caches/Homebrew/downloads/ffcfe300e2ad205387a2bdd27185d98d64621578d26d2f6ccbbcd091ff799fc8--python-3.7.4.mojave.bottle.tar.gz
==> Pouring python-3.7.4.mojave.bottle.tar.gz
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
  brew install python@2

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.4: 3,869 files, 60MB
pommeau:~ bouge$ pip --version
pip 19.1.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

$ pip list --outdated

Package Version Latest Type 
------- ------- ------ -----
pip     19.1.1  19.2.1 wheel

$ pip3 list --outdated

Package Version Latest Type 
------- ------- ------ -----
pip     19.1.1  19.2.1 wheel

$ pip install --upgrade pip

Collecting pip
  Using cached https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.1.1
    Uninstalling pip-19.1.1:
      Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.1

$ pip3 list --outdated

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 19.2.1 (/usr/local/lib/python3.7/site-packages), Requirement.parse('pip==19.1.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/python/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==19.1.1' distribution was not found and is required by the application

[defjaf]

Hi,

I admit I’m confused, since, as you say, the above is happening independent of pip_upgrade_outdated. Does your installation just not allow upgrading pip for some reason? Or am I missing the point? (I haven’t seen these issues, I don’t think.)

Andrew

[lucbouge]

NB: I am running macOS Mojave 10.14.5 (18F132) and Homebrew 2.1.8.

I made progress in understanding the bug.

After a fresh reinstallation of Python, the pip3 script can be accessed through 2 paths, which eventually end at the same physical file (i-node): /usr/local/Cellar/python/3.7.4/bin/pip3

$ brew reinstall python
...

$ ls -li /usr/local/Cellar/python/3.7.4/bin/pip3 /usr/local/bin/pip3

36634604 -rwxr-xr-x  1 bouge  staff   386B 24 jul 14:40 /usr/local/Cellar/python/3.7.4/bin/pip3*
36634660 lrwxr-xr-x  1 bouge  wheel    31B 24 jul 14:40 /usr/local/bin/pip3@ -> ../Cellar/python/3.7.4/bin/pip3

$ realpath /usr/local/Cellar/python/3.7.4/bin/pip3 /usr/local/bin/pip3
/usr/local/Cellar/python/3.7.4/bin/pip3
/usr/local/Cellar/python/3.7.4/bin/pip3

$ more /usr/local/Cellar/python/3.7.4/bin/pip3

#!/usr/local/opt/python/bin/python3.7
# EASY-INSTALL-ENTRY-SCRIPT: 'pip==19.1.1','console_scripts','pip3'
__requires__ = 'pip==19.1.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pip==19.1.1', 'console_scripts', 'pip3')()
    )

After the pip upgrading, it is no longer the case! 👎 A new physical file (i-node) has been created: /usr/local/bin/pip3 instead of the former /usr/local/bin/pip3, and the old /usr/local/bin/pip3, which was the same file as /usr/local/Cellar/python/3.7.4/bin/pip3 has not been updated.

Unfortunately, my PATH finds pip3 at /usr/local/opt/python/bin/pip3, according to the recommendation of brew info pip3, instead of /usr/local/bin/pip3. 👎👎👎👎👎

$ pip install --upgrade pip 

Collecting pip
  Using cached https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.1.1
    Uninstalling pip-19.1.1:
      Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.1

$ ls -li /usr/local/Cellar/python/3.7.4/bin/pip3 /usr/local/bin/pip3

36634604 -rwxr-xr-x  1 bouge  staff   386B 24 jul 14:40 /usr/local/Cellar/python/3.7.4/bin/pip3*
36636281 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 14:42 /usr/local/bin/pip3*

pommeau:~ bouge$ realpath /usr/local/Cellar/python/3.7.4/bin/pip3 /usr/local/bin/pip3

/usr/local/Cellar/python/3.7.4/bin/pip3
/usr/local/bin/pip3

$ brew info python

python: stable 3.7.4 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/3.7.4 (3,909 files, 60.6MB) *
  Poured from bottle on 2019-07-24 at 14:40:38
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/python.rb
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, openssl ✔, readline ✔, sqlite ✔, xz ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
Python has been installed as
  /usr/local/bin/python3

**Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin**

If you need Homebrew's Python 2.7 run
  brew install python@2

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 454,942 (30 days), 1,188,085 (90 days), 4,432,887 (365 days)
install_on_request: 222,671 (30 days), 575,343 (90 days), 2,441,170 (365 days)
build_error: 0 (30 days)

$ ll /usr/local/opt/python/libexec/bin/pip 

lrwxr-xr-x  1 bouge  staff    14B 24 jul 14:53 /usr/local/opt/python/libexec/bin/pip@ -> ../../bin/pip3

 $ realpath /usr/local/opt/python/libexec/bin/pip 

/usr/local/Cellar/python/3.7.4/bin/pip3

$ pommeau:~ bouge$ ls -li /usr/local/Cellar/python/3.7.4/bin/pip3

36634604 -rwxr-xr-x  1 bouge  staff   386B 24 jul 14:53 /usr/local/Cellar/python/3.7.4/bin/pip3*

$ more  /usr/local/Cellar/python/3.7.4/bin/pip3

#!/usr/local/opt/python/bin/python3.7
# EASY-INSTALL-ENTRY-SCRIPT: 'pip==19.1.1','console_scripts','pip3'
__requires__ = 'pip==19.1.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pip==19.1.1', 'console_scripts', 'pip3')()
    )

[lucbouge]

To summarize this long and complex post.

1. It is a problem with pip and homebrew, not with your pip_update_outdated script.

2. Your pip_update_outdated script should better refrain updating pip itself, and instead issue a warning message.

[defjaf]

Hi,

1. It is a problem with pip and homebrew, not with your pip_update_outdated script.

Agreed, I think. I use the python.org framework build, and don’t see this issue.

2. Your pip_update_outdated script should better refrain updating pip itself, and instead issue a warning message.

I don’t think it should — as above, I do want it to update pip for my framework build. You can always use pip_upgrade_outdated -x pip.

But this is a bug in homebrew python, right? Should it be reported there?

Andrew

[lucbouge]

Thank you for the answer.

I could find that the key is the PATH search priority. If you use /usr/local/bin/pip, then you are safe. The problem is with /usr/local/opt/python/libexec/bin/pip, which is badly recommend by brew info python.

I think it closes this issue. I filled a bug at the Homebrew site. Until it is solved, I use the following work-around: python3 -m pip install --upgrade pip; pip_upgrade_outdated

Thank you for your pip_upgrade_outdated script,

Luc.

[defjaf]

Dear Luc,

Glad we've resolved it, at least as far as pip_upgrad_outdated is concerned -- and I'm glad that you've found my code useful.

Let's hope homebrew fixes this issue!

Yours,

Andrew