Use CacheControl instead of custom cache code

* Deprecates the --download-cache option & removes the download
  cache code.
* Removes the in memory page cache on the index
* Uses CacheControl to cache all cacheable HTTP requests to the
  filesystem.
  * Properly handles CacheControl headers for unconditional
    caching.
  * Will use ETag and Last-Modified headers to attempt to do a
    conditional HTTP request to speed up cache misses and turn
    them into cache hits.
  * Removes some concurrency unsafe code in the download cache
    accesses.
  * Uses a Cache-Control request header to limit the maximum
    length of time a cache is valid for.
* Adds pip.appdirs to handle platform specific application
  directories such as cache, config, data, etc.

Author: dstufft

CHANGES.txt

@@ -37,6 +37,14 @@
 * Fixed :issue:`1775`. `pip wheel` wasn't building wheels for dependencies of
   editable requirements.
 
+* **DEPRECATION** ``pip install --download-cache`` and
+  ``pip wheel --download-cache`` command line flags have been deprecated and
+  the functionality removed. Since pip now automatically configures and uses
+  it's internal HTTP cache which supplants the ``--download-cache`` the
+  existing options have been made non functional but will still be accepted
+  until their removal in pip v1.8. For more information please see
+  https://pip.pypa.io/en/latest/reference/pip_install.html#caching
+
 
 **1.5.5 (2014-05-03)**
 

docs/reference/pip_install.rst

@@ -249,6 +249,36 @@ Starting with v1.3, pip provides SSL certificate verification over https, for th
 of providing secure, certified downloads from PyPI.
 
 
+.. _`Caching`:
+
+Caching
++++++++
+
+Starting with v1.6, pip provides an on by default cache which functions
+similarly to that of a web browser. While the cache is on by default and is
+designed do the right thing by default you can disable the cache and always
+access PyPI by utilizing the ``--no-cache-dir`` option.
+
+When making any HTTP request pip will first check it's local cache to determine
+if it has a suitable response stored for that request which has not expired. If
+it does then it simply returns that response and doesn't make the request.
+
+If it has a response stored, but it has expired, then it will attempt to make a
+conditional request to refresh the cache which will either return an empty
+response telling pip to simply use the cached item (and refresh the expiration
+timer) or it will return a whole new response which pip can then store in the
+cache.
+
+When storing items in the cache pip will respect the ``CacheControl`` header
+if it exists, or it will fall back to the ``Expires`` header if that exists.
+This allows pip to function as a browser would, and allows the index server
+to communicate to pip how long it is reasonable to cache any particular item.
+
+While this cache attempts to minimize network activity, it does not prevent
+network access all together. If you want a fast/local install solution that
+circumvents accessing PyPI, see :ref:`Fast & Local Installs`.
+
+
 Hash Verification
 +++++++++++++++++
 
@@ -264,26 +294,6 @@ It is not intended to provide security against tampering. For that,
 see :ref:`SSL Certificate Verification`
 
 
-Download Cache
-++++++++++++++
-
-pip offers a :ref:`--download-cache <install_--download-cache>` option for
-installs to prevent redundant downloads of archives from PyPI.
-
-The point of this cache is *not* to circumvent the index crawling process, but
-to *just* prevent redundant downloads.
-
-Items are stored in this cache based on the url the archive was found at, not
-simply the archive name.
-
-If you want a fast/local install solution that circumvents crawling PyPI, see
-the :ref:`Fast & Local Installs`.
-
-Like all options, :ref:`--download-cache <install_--download-cache>`, can also
-be set as an environment variable, or placed into the pip config file.  See the
-:ref:`Configuration` section.
-
-
 .. _`editable-installs`:
 
 "Editable" Installs

pip/appdirs.py

@@ -0,0 +1,138 @@
+"""
+This code was taken from https://github.com/ActiveState/appdirs and modified
+to suite our purposes.
+"""
+import os
+import sys
+
+from pip._vendor import six
+
+
+def user_cache_dir(appname):
+    r"""
+    Return full path to the user-specific cache dir for this application.
+
+        "appname" is the name of application.
+
+    Typical user cache directories are:
+        Mac OS X:   ~/Library/Caches/<AppName>
+        Unix:       ~/.cache/<AppName> (XDG default)
+        Windows:      C:\Users\<username>\AppData\Local\<AppName>\Cache
+
+    On Windows the only suggestion in the MSDN docs is that local settings go
+    in the `CSIDL_LOCAL_APPDATA` directory. This is identical to the
+    non-roaming app data dir (the default returned by `user_data_dir`). Apps
+    typically put cache data somewhere *under* the given dir here. Some
+    examples:
+        ...\Mozilla\Firefox\Profiles\<ProfileName>\Cache
+        ...\Acme\SuperApp\Cache\1.0
+
+    OPINION: This function appends "Cache" to the `CSIDL_LOCAL_APPDATA` value.
+    """
+    if sys.platform == "win32":
+        # Get the base path
+        path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))
+
+        # Add our app name and Cache directory to it
+        path = os.path.join(path, appname, "Cache")
+    elif sys.platform == "darwin":
+        # Get the base path
+        path = os.path.expanduser("~/Library/Caches")
+
+        # Add our app name to it
+        path = os.path.join(path, appname)
+    else:
+        # Get the base path
+        path = os.getenv("XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
+
+        # Add our app name to it
+        path = os.path.join(path, appname)
+
+    return path
+
+
+# -- Windows support functions --
+
+def _get_win_folder_from_registry(csidl_name):
+    """
+    This is a fallback technique at best. I'm not sure if using the
+    registry for this guarantees us the correct answer for all CSIDL_*
+    names.
+    """
+    import _winreg
+
+    shell_folder_name = {
+        "CSIDL_APPDATA": "AppData",
+        "CSIDL_COMMON_APPDATA": "Common AppData",
+        "CSIDL_LOCAL_APPDATA": "Local AppData",
+    }[csidl_name]
+
+    key = _winreg.OpenKey(
+        _winreg.HKEY_CURRENT_USER,
+        r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders"
+    )
+    directory, _type = _winreg.QueryValueEx(key, shell_folder_name)
+    return directory
+
+
+def _get_win_folder_with_pywin32(csidl_name):
+    from win32com.shell import shellcon, shell
+    directory = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)
+    # Try to make this a unicode path because SHGetFolderPath does
+    # not return unicode strings when there is unicode data in the
+    # path.
+    try:
+        directory = six.text_type(directory)
+
+        # Downgrade to short path name if have highbit chars. See
+        # <http://bugs.activestate.com/show_bug.cgi?id=85099>.
+        has_high_char = False
+        for c in directory:
+            if ord(c) > 255:
+                has_high_char = True
+                break
+        if has_high_char:
+            try:
+                import win32api
+                directory = win32api.GetShortPathName(directory)
+            except ImportError:
+                pass
+    except UnicodeError:
+        pass
+    return directory
+
+
+def _get_win_folder_with_ctypes(csidl_name):
+    csidl_const = {
+        "CSIDL_APPDATA": 26,
+        "CSIDL_COMMON_APPDATA": 35,
+        "CSIDL_LOCAL_APPDATA": 28,
+    }[csidl_name]
+
+    buf = ctypes.create_unicode_buffer(1024)
+    ctypes.windll.shell32.SHGetFolderPathW(None, csidl_const, None, 0, buf)
+
+    # Downgrade to short path name if have highbit chars. See
+    # <http://bugs.activestate.com/show_bug.cgi?id=85099>.
+    has_high_char = False
+    for c in buf:
+        if ord(c) > 255:
+            has_high_char = True
+            break
+    if has_high_char:
+        buf2 = ctypes.create_unicode_buffer(1024)
+        if ctypes.windll.kernel32.GetShortPathNameW(buf.value, buf2, 1024):
+            buf = buf2
+
+    return buf.value
+
+if sys.platform == "win32":
+    try:
+        import win32com.shell  # noqa
+        _get_win_folder = _get_win_folder_with_pywin32
+    except ImportError:
+        try:
+            import ctypes
+            _get_win_folder = _get_win_folder_with_ctypes
+        except ImportError:
+            _get_win_folder = _get_win_folder_from_registry

pip/basecommand.py

@@ -19,7 +19,7 @@
     SUCCESS, ERROR, UNKNOWN_ERROR, VIRTUALENV_NOT_FOUND,
     PREVIOUS_BUILD_DIR_ERROR,
 )
-from pip.util import get_prog
+from pip.util import get_prog, normalize_path
 
 
 __all__ = ['Command']
@@ -54,7 +54,10 @@ def __init__(self):
         self.parser.add_option_group(gen_opts)
 
     def _build_session(self, options):
-        session = PipSession(retries=options.retries)
+        session = PipSession(
+            cache=normalize_path(os.path.join(options.cache_dir, "http")),
+            retries=options.retries,
+        )
 
         # Handle custom ca-bundles from the user
         if options.cert:

pip/cmdoptions.py

@@ -9,7 +9,9 @@
 """
 import copy
 from optparse import OptionGroup, SUPPRESS_HELP, Option
-from pip.locations import build_prefix, default_log_file, src_prefix
+from pip.locations import (
+    USER_CACHE_DIR, build_prefix, default_log_file, src_prefix,
+)
 
 
 def make_option_group(group, parser):
@@ -322,12 +324,26 @@ def make(self):
           'find-links locations.'),
 )
 
+cache_dir = OptionMaker(
+    "--cache-dir",
+    dest="cache_dir",
+    default=USER_CACHE_DIR,
+    metavar="dir",
+    help="Store the cache data in <dir>."
+)
+
+no_cache = OptionMaker(
+    "--no-cache-dir",
+    dest="cache_dir",
+    action="store_false",
+    help="Disable the cache.",
+)
+
 download_cache = OptionMaker(
     '--download-cache',
     dest='download_cache',
-    metavar='dir',
     default=None,
-    help='Cache downloaded packages in <dir>.')
+    help=SUPPRESS_HELP)
 
 no_deps = OptionMaker(
     '--no-deps', '--no-dependencies',
@@ -396,6 +412,8 @@ def make(self):
         cert,
         client_cert,
         no_check_certificate,
+        cache_dir,
+        no_cache,
     ]
 }
 

pip/commands/install.py

@@ -248,6 +248,14 @@ def run(self, options, args):
             )
             index_urls += options.mirrors
 
+        if options.download_cache:
+            logger.deprecated(
+                "1.8",
+                "--download-cache has been deprecated and will be removed in "
+                " the future. Pip now automatically uses and configures its "
+                "cache."
+            )
+
         session = self._build_session(options)
 
         finder = self._build_package_finder(options, index_urls, session)
@@ -256,7 +264,6 @@ def run(self, options, args):
             build_dir=options.build_dir,
             src_dir=options.src_dir,
             download_dir=options.download_dir,
-            download_cache=options.download_cache,
             upgrade=options.upgrade,
             as_egg=options.as_egg,
             ignore_installed=options.ignore_installed,

pip/commands/wheel.py

@@ -144,6 +144,14 @@ def run(self, options, args):
             )
             index_urls += options.mirrors
 
+        if options.download_cache:
+            logger.deprecated(
+                "1.8",
+                "--download-cache has been deprecated and will be removed in "
+                " the future. Pip now automatically uses and configures its "
+                "cache."
+            )
+
         session = self._build_session(options)
 
         finder = PackageFinder(
@@ -162,7 +170,6 @@ def run(self, options, args):
             build_dir=options.build_dir,
             src_dir=options.src_dir,
             download_dir=None,
-            download_cache=options.download_cache,
             ignore_dependencies=options.ignore_dependencies,
             ignore_installed=True,
             session=session,