draft of new install instructions based on discussion in #1079

[qwcode]

in built form: http://qwcode.com/pip_docs/installing.html

[qwcode]

btw, I've pushed similar instructions up to the user guide. I will change according to feedback here.

https://python-packaging-user-guide.readthedocs.org/en/latest/setup.html

I would have logged it as PR for discussion, but:

• I felt confident that anything was better than what was there before
• pretty hard to get eyes on anything related to the user guide
• it's still WIP

[qwcode]

even though ez_setup.py is not used in the recommended procedure, it's a glaring issue that it doesn't use ssl cert verification during the download. big weakness. I will remove it from global the instructions and replace with a download/unpack/install sequence.

and will add a footnote about secure downloading for anywhere "download" is used

[ncoghlan]

Looks like a good interim solution to me, as well as a better guide for what we should offer in Python 3.4 (i.e. add pip support to pyvenv, rather than focusing on in-place bootstrapping)

[dstufft]

I just looked at what's available in the user guide, and one major thing concerns me. The recommendation to simply use an unpacked virtualenv instead of installing it somewhere (even if somewhere is the user packages) does not match what people do in real life. Further more I think it's unlikely that the vast bulk of people will do that in real life and I think that anyone who learns this method will eventually begin installing it somewhere for the convenience factor of having the command available instead of needing to call python ~/virtualenv1.10/virtualenv.py anytime they want to use it. This method also cleanly breaks tools like virtualenvwrapper which is recommended a lot throughout the community.

If things are broken that prevent being able to recommend installing it (either globally or via --user) we should fix those things instead of instructing users to hackishly work around the shortcomings.

@ncoghlan This probably isn't the appropiate venue for this discussion but I do believe that we need pip at the system level as well if we're going to include it with Python. pyvenv (and virtualenv) are not universally accepted and if we focus solely on pyvenv we don't solve the problem for people who want to use it outside of a virtual environment and leave the issue half solved.

[qwcode]

use an unpacked virtualenv [...] does not match what people do in real life [...] hackishly workaround

I don't know, I think using an unpacked virtualenv (or before it became unsupported, the older standalone pattern) is fairly common.

instead of needing to call python ~/virtualenv1.10/virtualenv.py anytime

Paul mentioned adding the unpacked vitualenv dir to your PATH, so the instructions could explain that (and then you'd just need to call virtualenv.py), although I'm honestly not sure how that works on windows and the generic shebang getting the right interpreter?

If things are broken that prevent being able to recommend installing it (either globally or via --user)

as for global installs, nothing is broken. the concern is the need for root/system tampering (largely voiced by Nick)

as for user installs, yes, stuff is probably broke right now, but realize that recommending a --user install procedure will certainly "not match what people do", and much stress will be had by all getting through a transition like that.

I just want something now (not after 1.5 is released in Dec) that won't be viewed as "circular" or considered "irresponsible".

I think I'll have to do a breakdown of the options and post to pypa-dev tomorrow.

[ncoghlan]

On 5 August 2013 16:09, Donald Stufft [email protected] wrote:

@ncoghlan https://github.com/ncoghlan This probably isn't the
appropiate venue for this discussion but I do believe that we need pip
at the system level as well if we're going to include it with Python.
pyvenv (and virtualenv) are not universally accepted and if we focus solely
on pyvenv we don't solve the problem for people who want to use it outside
of a virtual environment and leave the issue half solved.

Remember, the standard library solution is for beginners, and beginners
should not be using pip to install anything into their system Python.
Advanced users can still do a global install, and the instructions for that
are already reasonable enough for such users. So no, I don't believe a
standard library solution needs to endorse using pip for system package
installs. I'll reserve judgment until Brett puts a new PEP together (as we
may still offer "python -m getpip" at the system level, not just in a
virtualenv and as a pyvenv command line option), but don't be surprised if
that aspect effectively gets punted to 3.5.

[dstufft]

"System" level is actually a misnomer because Python isn't distributed at the system level by Python. Couching it in terms of a Linux distribution is pushing the concerns of a Linux distribution onto a cross platform language. I'm sure Linux packagers will rip pip out regardless of what we do and make it a separate package so for people installing it via their Linux package manager the proposal to include pip will likely have very little to no effect. So not only is it attempting to impose the constraints of one type of install on one platform on the broader language, it's type of install/platform that likely will ignore the PEP and undo it.

You say this is for beginners, however by advocating either python -m getpip or only installing pip inside of a virtualenv you're making things harder than they need to be for beginners because now they have to either know they need to run python -m getpip prior to installing something, or they need to understand what a pyvenv even is (and possibly why it differs from a virtualenv).

[dstufft]

@qwcode Do you have any data to back that up? I don't think i've ever seen anyone do that. Granted I don't follow virtualenv development or issues very much but I'd think if it was popular I'd have seen someone do it before?

[qwcode]

@dstufft mostly based on users being familiar with the old standalone pattern. virtualenv's install instructions used to be like 2 lines, with the 2nd line being "you can just grab the single file virtualenv.py and run it with python virtualenv.py."

[ncoghlan]

Window doesn't have a dependency system, but it does have a system
installer: MSI (MicroSoft Installer). pip doesn't register files in a way
that MSI understands, so it may cause problem when uninstalling Python, and
only avoids needing elevated privileges to install into site-packages
because the CPython installer installs Python in a way that doesn't comply
with Microsoft's platform guidelines, so site-packages ends up being world
writable.

More generally, I refuse to have CPython endorse supposedly
"cross-platform" instructions that are a bad idea on every platform
except Windows, and only work there because the CPython installer doesn't
comply with platform guidelines. If we installed into Program Files as the
guidelines say we should, then pip would need to be invoked via runas (or
otherwise acquire elevated privileges), just as it requires sudo on Linux
and Mac OS X.

The assumption to run with for the moment is that Python 3.4 will likely
support bootstrapping pip into pyvenv environments and may support
explicit bootstrapping outside virtual environments. Like PEP 439 before
them, both are contingent on the standard library gaining the ability to do
verified SSL with the system certs, which doesn't even have a PEP yet. If
you're worried about bootstrapping actually happening for Python 3.4,
asking Christian Heimes how to help with the SSL cert verification issue is
the best way to help address the single biggest blocking concern at this
point. Without that facility, any talk of bootstrapping anything is
completely hypothetical, because we can't do trusted downloads except
through a web browser (I guess opening the browser could be a fallback
option, but a pretty poor one).

[dstufft]

Firstly I'm not sure why bundling pip with Python itself vs only inside of a pyvenv somehow translates to endorsing installing into the system location. Especially after I've already talked to you about wanting to make sure that a globally installed pip plays nicely with the various systems and that installing globally is opt-in rather than opt-out. This might take the form of some sort of --system or --global flag but I haven't completely worked out my proposals to Python, Linux Distributions, and pip itself. Overall I think advocating for a different default than global installation is a far better path to tread than purposely negating one of the largest benefits that bundling pip has (The ability for someone new to install a package without needing to understand much else).

Secondly you state that it's a bad idea on every platform except windows which is untrue. It's generally only a bad idea on the system installed Python from *nix, and then only because a different tool expects to "own" that area of install which clashes with pip. Off the top of my head here are places where it's perfectly fine, and even preferable to install globally with pip:

• Windows MSI installer from Python.org
• OSX Installer from Python.org
• Homebrew on OSX
• Python installed by default on OSX
  • This one is somewhat iffy, but as far as I know no system utility uses it and using easy_install or pip is the only way to install anything into it.
• Python installed outside of the system package manager on Linux (or really any platform)
• Python installed via Conda or a Conda like system
  • This one can arguably be covered by Conda installing pip itself

Of those 6 off the top of my head 3 of them are installation methods that are most certainly used by beginners and a fourth and possibly fifth is one that is very likely used by beginners. Basically it's a perfectly fine, if not the preferred or only option for installation any time you have a version of Python not controlled by a system level package management tool. I think you are letting your close proximity to Linux system packages influence your viewpoint and the fact that in that one specific scenario it can be a bad idea to install globally using pip prevent you from seeing the larger picture.

Quite frankly I find the bit about Windows bordering on FUD. You state that it avoids needing elevated privileges but I don't think that is a thing that even matters in this context. If it's installed to Program Files, yes it would need elevated privs but all that means is it's on the same footing as the *nix's in which it would still not having any bearing on whether or not it's a good idea to install something into site-packages. As to the not registering with MSI, installing a Python package via pip is explicitly outside of the realm of MSI, just the same as any other software you'd install on Windows that then installed other software. It is true that those files would be left behind during an uninstall it doesn't prevent or cause any sort of trouble when uninstalling Python (and Python itself leaves files behind).

Can you point out any other instances where using pip to install to site-packages is a bad idea? Because literally the only one I can come up with is *nix, and not just *nix, but specifically *nix where the Python in question is installed and "owned" by the system package manager.

Thirdly, bootstrapping of any kind that fetches and installs from the internet on end user machines is likely to be a bad idea. Not only do you place a requirement for networking (of which there may be none) and a fully functioning SSL stack (which is hardly guaranteed as we learned in pip) in order to install the packages (which are not required to be fetched over the internet), but you also by requirement need to implement your own little mini pip in order to do so properly. Thinking that CPython is both going to get it right immediately, and is going to continue to get it right as things evolve with a version that will not have anywhere near the amount of testing and real world use as pip itself is hubris. Pip recognized that building a mini pip is a bad idea and the get-pip.py script bundles an entire copy of pip which it uses to pip install pip.

[ncoghlan]

Bundle pip with CPython. Install CPython. Upgrade pip. Upgrade CPython to a
new maintenance release. What happens now?

The only things I genuinely don't like at this point are:

• recommending bootstrapping pip over using a virtualenv in a Python you
  didn't install yourself
• having the pip install "owned" by the python.org installers. It must be
  clearly owned by pip, and upgraded through pip's mechanisms.

If pip can get to the point of having a zip based self-installer that can
be run with "python pip-bootstrap.pyz" and uses the system certs (rather
than needing to bundle certs, which would then be cached for every in old
CPython installers), then I would be fine with the idea of bundling that
and giving the Mac OS X and Windows installers an option to run it
following the install, as well as a second dependent option to immediately
upgrade it from PyPI. No such self-installer currently exists though, so
its a wish list item that isn't currently available as an option to be
considered.

Are you sure you don't want to write the bundling/explicit bootstrapping
PEP?

[ncoghlan]

I'll note, though, that the contents of a "pip-bootstrap.pyz" and a
getpip package in the standard library could be identical, so they're
functionally equivalent solutions from a system installation
perspective, but "getpip" offers the ability to easily provide pip in
every pyvenv as well.

[ncoghlan]

Perhaps that's the part that I haven't been clear on? That I'm
actually OK with the idea of "python -m getpip" being a package that
has a complete vendored copy of pip in it that it either uses to get a
more recent version from PyPI, or else just installs directly as if
from a wheel file?

[dstufft]

I just talked to Paul to ask him if he's had a chance to work on it or if he wanted me to take a shot and I believe I'm going to write it up this week hopefully.

Does Python support installing a maintenance release over top of an installed release? Either way as far as I know Python doesn't do anything with the site-packages directory (at least on Windows, I use Homebrew for OSX which does things differently and apt on Ubuntu). So if installing a maintenance overtop of an existing one is a supported thing, then as long as whatever lays pip down inside of site-packages checks for the existence of it already installed pip and just NOPs it that should work fine off the top of my head.

FWIW I'm all for recommending (and even guiding) users towards --user and virtualenv/pyvenv. I just don't want the PEP to be less useful than it could be to the widest group of people because we got prescriptive instead of descriptive.

In the midst of writing this I got your message about python -m getpip essentially being a bundled pip, so I'm going to roll with that.

I think the ideal method is python -m getpip is included and has a bundled version of pip (this should ideally be updated relatively often, perhaps anytime pip releases?). It will attempt to fetch the latest version from PyPI and failing that will fallback to using it's bundled copy as well as having flags to force network or bundled installation. Then the installers for Windows and OSX should be modified to (optionally? Opt Out) run python -m getpip during install time. We should then recommend that anybody packaging/creating an installer for Python should use whatever mechanism to make it easy to have pip installed by default. On Linux this would probably correspond to a "Recommends" dependency (I think? That's the one that installs by default, but can be disabled and won't complain if it's missing).

This will:

• Provide pip by default on Windows and OSX where the installers from Python are most widely used
• Provide recommendations to other distributors to include pip in some form by default
• Provide a method of installing pip that does not depend on the network, but will use it if available
• Provide an easy method of fetching pip if, for whatever reason, you don't have it available
  • Linux deciding not to depend on python-pip
  • Someone uninstalling pip
  • Something else?
• Provides a simple method that can be used inside of a pyvenv as well
• Hopefully put this issue to rest and we can find something new to argue about

It does still leave part of the question on the table of upgrading Python and what should happen, should it issue another python -m getpip which will do the network/fallback dance again to see if there is a newer version than what's installed? Or should it simple check that a version of pip is installed and be happy with that.

One of the things I'm really pushing for is that if you install Python then pip install <foo> should just work without you needing to do something else. Obviously there might be situations where that doesn't hold true since we are not adding pip itself to the standard library and we can't control downstream so a method to fetch it isn't a bad thing. I just feel pretty strongly that having all users need to do an explicit "actually install pip" step in the common case makes the proposal far less effective at the pragmatic goal of making things better for users.

[ncoghlan]

That sounds really good - I'll let Brett know he's off the hook after all
(he'd offered to tackle it if nobody else volunteered before PyCon CA)

[qwcode]

distutils-sig discussions are like a gas that will fill in any space or medium it can find.

[qwcode]

closing this per discussion on pypa-dev: https://groups.google.com/forum/#!topic/pypa-dev/A7XwxElwFI8