pip doesn't use proxy for search command

[albgus]

I'm attempting to use pip to install packages to a python virtualenv. I have however discovered that when trying to search for commands, the proxy is not used. running for example pip search pika fails while pip install pika works fine.

I have set the environment variables http_proxy, https_proxy and no_proxy accordingly and I have tried to set the --proxy option with no success. It does however ask for a password if I specify a username but no password, so at least it acknowledges the options (the proxy is anonymous login)

I run into this issue on RedHat 6.4, with the default Python 2.6.6, and Python 2.7.5 from the tarball distribution.

Here are the logs:

Traceback (most recent call last):
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 38, in run
    pypi_hits = self.search(query, index_url)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 52, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 1112, in connect
    sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
timeout: timed out

Storing complete log in /home/gusson/.pip/pip.log

If I run a strace on the process it stops on this message, indicating that it tries to contact the source directly without going via the proxy, thus gets blocked by the firewall.

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("199.27.73.185")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, 15000

[monkeypants]

pip search uses xmlrpclib, which isn't natively proxy aware. pip install uses urllib2, which is (uses the HTTP_PROXY environment variable etc).

There are various examples of people creating a urllib2 transport and tunneling xmlrpclib over it, see for example https://gist.github.com/nathforge/980961 .There is even something like this happening in vendor/distlib/util.py.

Simply patching commands/search.py to use code from nathforge's gist didn't work though, possibly because pip search it's using xmlrpclib from pip.backwardcompat rather than the native one.

[kkdd]

The replacement of xmlrpclib.ServerProxy() in search.py by the xmlrpclib_ServerProxy() defined below works in my environment of Python 2.7.5 and pip 1.4.1.

import os

def get_proxy():
    proxy = os.getenv('http_proxy')
    if proxy and proxy.startswith("http://"):
        proxy = proxy[len('http://'):]
    return proxy

def xmlrpclib_ServerProxy(index_url):
    if get_proxy():
        p = ProxiedTransport()
        return xmlrpclib.ServerProxy(index_url, transport=p)
    else:
        return xmlrpclib.ServerProxy(index_url)

import httplib

class ProxiedTransport(xmlrpclib.Transport):
    def make_connection(self, host):
        self.realhost = host
        return httplib.HTTPConnection(get_proxy())
    def send_request(self, connection, handler, request_body):
        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
    def send_host(self, connection, host):
        connection.putheader('Host', self.realhost)

[kkdd]

The above version of ProxiedTransport() used httplib. In contrast, the following uses urllib2. Also works in my environment of Python 2.7.5 and pip 1.4.1.

import urllib2

class ProxiedTransport(xmlrpclib.Transport):
    SCHEME = 'https'
    def __init__(self, *args, **argd):
        xmlrpclib.Transport.__init__(self, *args, **argd)
        handlers = []
        proxy = urllib2.ProxyHandler({'http':get_proxy()})
        handlers.append(proxy)
        self.opener = urllib2.build_opener(*handlers)
    def request(self, host, handler, request_body, verbose=0):
        self.verbose = 0
        scheme = self.SCHEME
        url = '%(scheme)s://%(host)s%(handler)s' % locals()
        req = urllib2.Request(url, data=request_body,
                              headers={'Content-Type':'text/xml'})
        fp = self.opener.open(req)
        return self.parse_response(fp)

[drakeguan]

To help make this issue proceed, there is a pull request, #1180, which provides a HttpProxyTransport for xmlrpclib.ServerProxy to work perfectly with proxy. I have tested it and it worked well here. That pull request might be one of options for this issues?

[monkeypants]

I think that fixes the bug (--proxy ignored) and provides a work arround (pass in --proxy).

It would be nice if pip fell back to use HTTP_PROXY environment variable too, but not essential.

[ash211]

Observed in pip 1.3.1 on CentOS 6.4 with EPEL's packages.

Works:

pip install $package

Doesn't work:

pip search $package

[spuk-]

You don't need to set the proxy handler manually for urllib2 as it automatically detects *_proxy env vars. Also, the code above didn't work for me, didn't care to check throughly why, but it simply wasn't using the proxy although get_proxy() appeared to be finding it. Simplified patch here: https://gist.github.com/spuk-/11256147#file-pip-search-urllib2

[jrosiek]

spuk's solutions is great! - simple and working. Please incorporate it into next release, thanks!

[Ivoz]

Relevant PRs #932, #1180

Alternate fix would be through #395

[Ivoz]

Should be fixed in 1.6 through #1902 as above