At first glance, pip and virtualenv installation instructions form an infinite loop

[ncoghlan]

It was pointed out to me on Twitter that pip currently says "use virtualenv to install pip" as it's preferred option, while the first suggestion in the virtualenv installation instructions says "use pip to install virtualenv".

The bootstrap script is mentioned well down towards the bottom of the page, and using the system package manager (for Linux users) isn't mentioned at all.

[ncoghlan]

Working on a patch for the docs :)

[dstufft]

I wonder how hard it would be to provide third party repos that package up to date pips for all the popular linuxes.

[ncoghlan]

If you bootstrap virtualenv, it doesn't really matter how old the system pip is.

[jezdez]

@ncoghlan Sorry but we don't want to recommend installing the system packages, since frankly they are often out of date. It increases the maintenance burden and in case of pip is even insecure.

[qwcode]

"currently says "use virtualenv to install pip" as it's preferred option,
while virtualenv says "use pip to install virtualenv""

it's only "circular" if you skim imo. the pip instructions offer virtualenv or global instructions. the virtualenv instructions offer 3 ways to install or use virtualenv. only 1 of the 3 involves using pip to install it.

but feel free to make it less vulnerable to skimming.

[ncoghlan]

Pull request at #1081

Hit the distros with a security bug if you want them to fix the system packages.

[ncoghlan]

But we need to get away from recommending the use of Python specific tools to manipulate system installs.

[ncoghlan]

The answer is not "Don't use system packages", it has to be "Why aren't distro packages updated in a timely manner?"

[jezdez]

@ncoghlan So you're saying to fix our "problems" we have to depend on numerous system packagers instead our own? Thanks but no thanks.

[jezdez]

Because it's a lot of work. We've had those terms in the docs a while ago, trust me. But it was a nightmare since people were using stone age old versions all the time. So we opted to include pip in virtualenv and promote virtualenv as a main way to handle Python packaging.

[qwcode]

despite being somewhat defensive about the "it's circular" line of attack (because it's not really), I do like the idea of a simple install sequence that people get used to vs a decision the user makes up front about global vs virtualenv.

1. get setuptools (this goes away in pip-1.5 anyway hopefully with "autosetuptools")
2. get pip
3. use pip to get virtualenv

not sure about the --user defaults though. we need to analyze what bugs we have first.

[ncoghlan]

I'm saying you're telling Linux beginners to fuck with their system Python install, and that's completely irresponsible.

[ncoghlan]

Yes, you can use pip to install packages globally on Linux, but it's easy to screw things up if you do that, and unless you know what you're doing, you may fundamentally break your system (especially on Fedora et al, where many system tools, including the system package manager, are also written in Python).

For cross-platform bootstrapping, get-pip.py needs to be front and centre, and it needs to be run without touching the system Python directories.

[jezdez]

@ncoghlan Hold on, let's take a step back about calling anything irresponsible here. How is it our responsibility to keep distro packages up to date. It's a huge pain and involves impossible bureaucracy to even be informed of changes that are done there that could break pip or virtualenv (e.g. Ubuntu/Debian's dist-packages), let alone be involved in the update cycles.

Just as @qwcode siad I'm afraid of using --user as the primary way to installing things simply because it was prone to breakage for a long time and is not a "common" enough pattern in my experience. I just tried to use it on my Mac and it didn't work for example.

[ncoghlan]

It's not your responsibility to keep distro packages up to date, it's your responsibility to give novice users instructions that don't run the risk of breaking their systems, and don't encourage them to run arbitrary code from the internet with elevated privileges.

Since per user installs that don't require elevated privileges apparently don't work, then I'll update the pull request accordingly, but it means the status quo is genuinely bad, not just poorly documented.

[jezdez]

Thank you for your feedback, @ncoghlan. We're aware that we don't give beginners a step by step guide that securely and risk-free allows them to use pip and virtualenv. I would suggest to separate the installation instructions into sections per distribution. Then link to the appropriate documentation of those systems how to install Python software. In doubt ez_setup.py and get-pip.py should work but again in practice this needs to be tested with each and every of those systems to be certain. I'd be happy if you could help us do that.

[qwcode]

if we can get something like "autosetuptools" (#1069) and some more user fixes into 1.5 (user installs do work better than the used to, but still have some bugs), then we can certainly have a simpler install story.

[pfmoore]

I agree with providing a simple, safe, install sequence. In my view, that sequence should be (and this is what I use personally):

1. Get virtualenv.py and the virtualenv_support directory (this step can and should be made more straightforward - a get-virtualenv script, for example).
2. Put it somewhere on your PATH.
3. Use virtualenv.py to vreate virtualenvs with pip and setuptools installed.
4. Never, ever, mess with your system Python. Always use virtualenvs.

When Python ships with pip we can have a different discussion, but currently, I believe we should only recommend using pip in a virtualenv.

Sure, we can document how the other approaches would work. But let's not recommend them, and let's say "advanced users only - use at your own risk".

[jezdez]

@pfmoore That's the sequence I use personally and which is how virtualenv.py was supposed to be used originally. I currently use easy_install to install virtualenv first due to the missing bootstrapping step, but I could see a get-virtualenv.py happening, too.

Are you suggesting to leave out system python package managers completely out of the novice docs? Maybe we should have a "quickstart" guide and an "advanced" guide in a separate document, so that beginners don't accidentally skim it and use the advanced technique.

[ncoghlan]

I've calmed down now, and am trying to figure out how to actually use get-pip.py on Windows. Oh my lord... how hard can an OS make it to save a file from the internet with the right extension?

[ncoghlan]

I tweaked the description to be clear that it's the "first impression" that forms an infinite loop, rather than the instructions actually being circular.

I closed my pull request, since I think what Jannis and Paul are talking about above sounds like a more promising path towards my preferred result of "leave the system Python alone until you know what you're doing".

However, I still updated that pull request with the results of my experience actually bootstrapping the Python installations on my gaming PC using Internet Explorer to do the download.

On the positive side: aside from the known "scripts aren't on PATH by default under Windows" problem both ez_setup.py and get-pip.py worked flawlessly once I had them downloaded.

On the negative side: Windows makes "download this script" an annoyingly difficult instruction to follow, and that's going to affect any virtualenv bootstrapping instructions, too :(

[ncoghlan]

And as an added realisation... you don't need to use "runas" to do system wide package installs on Windows, but that's only because the Windows installers don't put site-packages inside a protected operating system directory. This has upsides (Windows users don't have to grant the installers for Python packages extra privileges, so those are less dangerous) and downside (we're not taking any advantage of the additional OS levels protections for the intepreter and standard library). At this point in time, it's probably a net win that CPython doesn't comply with the platform standards...

[pfmoore]

@jezdez the only thing we need to watch is that people also install virtualenv_support. The completely standalone approach of just using virtualenv.py and downloading the rest was dropped because of the security issues, but the standalone-with-support-dir approach is still the best install method IMO (it's just lacking in a helper script to make it beginner-friendly).

And yes, based on @ncoghlan's comments, I'm saying that it sounds like we're doing more harm than good by even mentioning the system Python. (It's less of an issue on Windows, but I've no problem with saying the same thing there just for consistency).

@ncoghlan Why don't we make "python -m urllib " do the right thing and put this whole problem to bed once and for all? (I deleted the dig about how Windows is just being more secure than Unix, by avoiding making it easy to execute arbitrary code from the Internet - I don't think you're that much calmed down :-))

[pfmoore]

@ncoghlan FWIW, you can download a file in Powershell using

(new-object System.Net.WebClient).DownloadFile(<url>, <path>)

But there is a gotcha, the path has to be absolute for .NET stupidity reasons.... So "just install curl, it's not that hard" is still my honest answer to the file download issue :-)

[ncoghlan]

@pfmoore: I actually had the same thought ("Wait, I'm complaining about Windows doing something security related right by making it hard for users to do something that is quite dodgy from a security perspective?"), but then I remembered it's only downloading and executing text files through an interpreter that is a challenge. The whole process of running downloaded code is far more streamlined with an exe or an msi installer than it is with "download and run this script". It's just that using a prebuilt exe doesn't really work for this situation as it makes it too painful to find the Python installation. (Tangent: I was also lamenting the fact that one of the things causing problems is the madness of hiding file extensions by default while still using them to control behaviour, but then I realised that *nix isn't really any better in that regard since it ignores file extensions when executing scripts in favour of the shebang line inside the file)

I'm wary of relying on python -m urllib because it still doesn't do cert verification by default in 3.3, doesn't even support it in 2.7, and we hope to have a better pip bootstrapping solution baked in to the 3.4 installers (Brett Cannon has offered to help push that forward after PyCon Canada next month).

Accordingly, providing instructions on how to install a Windows version of curl as part of the Windows bootstrapping instructions is actually starting to sound pretty good to me :)

[pcarbonn]

Install curl on Windows ? Why not install pip for Windows instead ? It's a tiny python package manager for Windows, which installs pip automatically (using the ez-setup.py + get-pi.py route).

[qwcode]

I don't see how we can really go with the "use pip only in virtualenv" motto. It can be recommended, as it is now (and maybe add a virtualenv bootstrap like @jezdez mentioned), but too many people will be alienated by this who don't use virtualenv (e.g. people who maintain additional non-system python installs to underly their apps)

[dholth]

The autosetuptools branch works as a runnable zip so instead of get-pip.py you could have got-pip.zip, use that to get virtualenv, and then use that to install pip again. Issues include finding the correct Python to run it with and keeping it up to date. The whole setuptools - virtualenv - pip deal is a bit of a self-eating Ouroboros of recursive dependency.

[qwcode]

The autosetuptools branch works as a runnable zip so instead of get-pip.py you could have got-pip.zip

nice.

[ncoghlan]

On 25 Jul 2013 05:25, "Marcus Smith" [email protected] wrote:

I don't see how we can really go with the "use pip only in virtualenv"
motto. It can be recommended, as it is now (and maybe add a virtualenv
bootstrap like @jezdez mentioned), but too many people will be alienated by
this who don't use virtualenv (e.g. people who maintained non-system base
installs to underly their apps)

Sure, it's only about treating "add pip directly to the current Python" as
an advanced use case, while for the beginner case we focus on
bootstrapping virtualenv in earlier versions and getting pyvenv to
bootstrap pip in later versions.

[qwcode]

so, in thinking about making paul's 1-4 scenario the standard instrucitons...

about this part "Use virtualenv.py to create virtualenvs"

is that:

1. telling people to use virtualenv.py as an executable
   or
2. telling people to do "pythonX.X virtualenv.py"

I like #2, because it's clearer what's happening, and makes it clear they can use virtualenv.py against any installed python.

[pfmoore]

@qwcode I like #2 because it's more explicit, and after all these instructions are intended for beginners, so explicit is important.

[qwcode]

logged PR for this #1103

[qwcode]

per https://groups.google.com/forum/#!topic/pypa-dev/A7XwxElwFI8
for now, I have at least tried to remove the circularity.
also, setuptools instructions are included. no link out.

9242dbf
http://www.pip-installer.org/en/latest/installing.html