Remove setuptools requirements in pip itself

[dstufft]

The past two released of pip have had issues relating to the setuptools dependency. Ideally pip itself will no longer have any sort of dependency on setuptools, and the only actual dependency for setuptools will exist when a project uses setup.py.

[coderanger]

+INT_MAX. As the maintainer the Chef tooling for pip I have huge issues every time pip changes its setuptools requirements, and the result is usually a non-functional system as you can't use pip to install the correct versions of things once it is broken.

[qwcode]

related: #1049
contains a lot of pro/con analysis.

[qwcode]

there is no PEP426/MEBs yet, so unclear what your plan is for this?

[dstufft]

I mean that pip can depend on setuptools for actually executing a setup.py (e.g. just assume that the setup.py will be using setuptools), but that pip itself no longer depends on setuptools. This is important for Wheel installs in particular.

[qwcode]

right, but how are you planning on implementing this idea of "pip can depend on setuptools for actually executing a setup.py". daniel's "autosetuptools", was one idea, where we vendor pkg_resources and we automatically install setuptools when we need to, but that has concerns.

[dstufft]

I don't know that we actually need to do anything. Pip will still require you to have setuptools installed in order to install from sdists. So for most people there will still be a "dependency" on setuptools. The difference being we've firewalled it off to only the setup.py execution so we don't run into problems where a broken setuptools means a broken pip where you have to manually install setuptools to fix it.

[pfmoore]

So, what I understand you to be saying is that we vendor pkg_resources, and redirect all of our internal calls to pkg_resources to the vendored version. We could go further and remove/migrate some of those calls to avoid pkg_resources (possibly in favour of the already-vendored distlib) but this is optional, and probably not worth the risk.

Users still need setuptools installed to install from sdist (because we inject setuptools into the setup.py invocation) but that's orthogonal to our internal use of pkg_resources.

Is that a correct interpretation?

If so, +1 from me.

[dstufft]

Yup that's basically what I mean.

[qwcode]

we talked over IRC as well.
yes, +1 on vendoring pkg_resources.
daniel did the vendoring in PR #1069, but that includes the automatic install of setuptools, which I don't think anyone wants to do right now.

[dstufft]

Maybe we should consider this for 1.5.1. There has been a number of folks grumbling about the setuptools 0.8 requirement such as http://stackoverflow.com/questions/20905350/latest-pip-fails-with-requires-setuptools-0-8-for-dist-info

[qwcode]

yea, I guess more people than we might have imagined are using a mixture of old setuptools/distribute + new pip.
although in the SO, the answer given is fine. the author thinking he has setuptools 2.0.2 must be for another python.

[pfmoore]

I just did a very simple proof of concept patch (replace "import pkg_resources" with "from pip._vendor import pkg_resources") and it seems to work for some basic tests.

Also, with this patch, you can (almost!) create a standalone zipfile of pip which can be run from a bare Python interpreter. There's a problem with finding the SSL certificate (see below) but that can probably be sorted with a bit of effort.

>.\ww\Scripts\python.exe .\pip.zip install setuptools pip
Downloading/unpacking setuptools
  Could not fetch URL https://pypi.python.org/simple/setuptools/: There was a problem confirming the ssl certificate: [Errno 2] No such file or directory

[pfmoore]

The following __main__.py fixes the SSL issue, but vendoring setuptools is harder than the proof of concept above (no surprise :-)). I'm putting a copy of my __main__.py here anyway in case it's worth adding to pip as part of this change.

import sys, os
from pip.runner import run
from pkgutil import get_data
from tempfile import TemporaryDirectory

if __name__ == '__main__':
    cert_data = get_data('pip._vendor.requests', 'cacert.pem')
    with TemporaryDirectory() as td:
        tempname = os.path.join(td, 'cacert.pem')
        with open(tempname, 'wb') as tf:
            tf.write(cert_data)
        os.environ['REQUESTS_CA_BUNDLE'] = tempname
        exit = run()
    if exit:
        sys.exit(exit)

[dholth]

Right on. When I did it, it was pretty much as simple as "from pip.vendor import pkg_resources". This takes care of pip's need to enumerate installed packages.

I also remember a special case or two intended to help with upgrading distribute or setuptools that kindof breaks here.

On the other hand setuptools is only needed by the setup.py sub-processes and pip shouldn't vendor it.

It will be more complicated to fix pip to get smarter about setup/build dependencies but that is a totally different issue. I will be very happy when "pip install setuptools" always works.

[pfmoore]

pkg_resources also uses _markerlib, so that needs to be vendored, and pkg_resources amended, as well.

[dholth]

I think pkg_resources also has its own markers implementation, used in a different place, that came later. But vendoring pkg_resources' vendored markerlib is fairly easy. See old patch :-)

[dstufft]

Moving this to 1.5.1

[dstufft]

$ python -m pip

Usage:
  /Users/dstufft/.virtualenvs/650292f54bdb66c8/bin/python -m pip <command> [options]

Commands:
  install                     Install packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  zip                         Zip individual packages.
  unzip                       Unzip individual packages.
  bundle                      Create pybundles.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output.
  --log-file <path>           Path to a verbose non-appending log, that only logs failures. This log
                              is active by default at /Users/dstufft/.pip/pip.log.
  --log <path>                Path to a verbose appending log. This log is inactive by default.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --timeout <sec>             Set the socket timeout (default 15.0 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe,
                              (b)ackup.
  --cert <path>               Path to alternate CA bundle.

$ python -m pip install requests
Downloading/unpacking requests
  Using download cache from /Users/dstufft/.pip/cache/https%3A%2F%2Fpreview-pypi.python.org%2Fpackages%2F2.7%2Fr%2Frequests%2Frequests-2.1.0-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests
Cleaning up...

 $ python -m pip list
pip (1.5, /Users/dstufft/projects/pip)
requests (2.1.0)
wsgiref (0.1.2)

[dstufft]

The above are in a virtualenv where I've uninstalled setuptools. I had to use python -m pip because I installed using setup.py instead of Wheel so the generated pip script depended on pkg_resources (which isn't the case for Wheel installs).

[dstufft]

Here's what happens when you force it to use a sdist:

$ python -m pip install --no-use-wheel requests
Downloading/unpacking requests
  Using download cache from /Users/dstufft/.pip/cache/https%3A%2F%2Fpreview-pypi.python.org%2Fpackages%2Fsource%2Fr%2Frequests%2Frequests-2.1.0.tar.gz
  Running setup.py (path:/Users/dstufft/.virtualenvs/650292f54bdb66c8/build/requests/setup.py) egg_info for package requests
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
    ImportError: No module named setuptools.command
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 3, in <module>

ImportError: No module named setuptools.command

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /Users/dstufft/.virtualenvs/650292f54bdb66c8/build/requests
Storing debug log for failure in /Users/dstufft/.pip/pip.log

[dstufft]

I'll probably explicitly check for setuptools to be installed and give a better error message when installing from sdist, but it's working.

[pfmoore]

Nice. Can you put this up as a PR when it's ready, and I'll have a play with it?

[dstufft]

Yup!