Merge branch 'master' into add-use-user-site-to-preparer

Author: pradyunsg

.github/no-response.yml

@@ -1,5 +1,5 @@
 # Number of days of inactivity before issue is closed for lack of response
-daysUntilClose: 30
+daysUntilClose: 15
 # Label requiring a response
 responseRequiredLabel: "S: awaiting response"
 # Comment to post when closing an Issue for lack of response. Set to `false` to disable

docs/html/development/architecture/package-finding.rst

@@ -15,17 +15,16 @@ Overview
 Here is a rough description of the process that pip uses to choose what
 file to download for a package, given a requirement:
 
-1. Access the various network and file system locations configured for pip
-   that contain package files. These locations can include, for example,
-   pip's :ref:`--index-url <--index-url>` (with default
-   https://pypi.org/simple/ ) and any configured
-   :ref:`--extra-index-url <--extra-index-url>` locations.
-   Each of these locations is a `PEP 503`_ "simple repository" page, which
-   is an HTML page of anchor links.
-2. Collect together all of the links (e.g. by parsing the anchor links
-   from the HTML pages) and create ``Link`` objects from each of these.
-   The :ref:`LinkCollector <link-collector-class>` class is responsible
-   for both this step and the previous.
+1. Collect together the various network and file system locations containing
+   project package files. These locations are derived, for example, from pip's
+   :ref:`--index-url <--index-url>` (with default https://pypi.org/simple/ )
+   setting and any configured :ref:`--extra-index-url <--extra-index-url>`
+   locations. Each of the project page URL's is an HTML page of anchor links,
+   as defined in `PEP 503`_, the "Simple Repository API."
+2. For each project page URL, fetch the HTML and parse out the anchor links,
+   creating a ``Link`` object from each one. The :ref:`LinkCollector
+   <link-collector-class>` class is responsible for both the previous step
+   and fetching the HTML over the network.
 3. Determine which of the links are minimally relevant, using the
    :ref:`LinkEvaluator <link-evaluator-class>` class.  Create an
    ``InstallationCandidate`` object (aka candidate for install) for each
@@ -111,6 +110,12 @@ One of ``PackageFinder``'s main top-level methods is
    class's ``compute_best_candidate()`` method on the return value of
    ``find_all_candidates()``. This corresponds to steps 4-5 of the Overview.
 
+``PackageFinder`` also has a ``process_project_url()`` method (called by
+``find_best_candidate()``) to process a `PEP 503`_ "simple repository"
+project page. This method fetches and parses the HTML from a PEP 503 project
+page URL, extracts the anchor elements and creates ``Link`` objects from
+them, and then evaluates those links.
+
 
 .. _link-collector-class:
 
@@ -119,12 +124,8 @@ The ``LinkCollector`` class
 
 The :ref:`LinkCollector <link-collector-class>` class is the class
 responsible for collecting the raw list of "links" to package files
-(represented as ``Link`` objects). An instance of the class accesses the
-various `PEP 503`_ HTML "simple repository" pages, parses their HTML,
-extracts the links from the anchor elements, and creates ``Link`` objects
-from that information. The ``LinkCollector`` class is "unintelligent" in that
-it doesn't do any evaluation of whether the links are relevant to the
-original requirement; it just collects them.
+(represented as ``Link`` objects) from file system locations, as well as the
+`PEP 503`_ project page URL's that ``PackageFinder`` should access.
 
 The ``LinkCollector`` class takes into account the user's :ref:`--find-links
 <--find-links>`, :ref:`--extra-index-url <--extra-index-url>`, and related
@@ -133,6 +134,10 @@ method is the ``collect_links()`` method. The :ref:`PackageFinder
 <package-finder-class>` class invokes this method as the first step of its
 ``find_all_candidates()`` method.
 
+``LinkCollector`` also has a ``fetch_page()`` method to fetch the HTML from a
+project page URL. This method is "unintelligent" in that it doesn't parse the
+HTML.
+
 The ``LinkCollector`` class is the only class in the ``index`` sub-package that
 makes network requests and is the only class in the sub-package that depends
 directly on ``PipSession``, which stores pip's configuration options and

docs/html/development/release-process.rst

@@ -80,13 +80,13 @@ Creating a new release
 ----------------------
 
 #. Checkout the current pip ``master`` branch.
-#. Ensure you have the latest ``nox`` and ``twine`` installed.
+#. Ensure you have the latest ``nox`` installed.
 #. Prepare for release using ``nox -s prepare-release -- YY.N``.
    This will update the relevant files and tag the correct commit.
 #. Build the release artifacts using ``nox -s build-release -- YY.N``.
    This will checkout the tag, generate the distribution files to be
    uploaded and checkout the master branch again.
-#. Upload the distribution files to PyPI using ``twine upload dist/*``.
+#. Upload the release to PyPI using ``nox -s upload-release -- YY.N``.
 #. Push all of the changes including the tag.
 #. Regenerate the ``get-pip.py`` script in the `get-pip repository`_ (as
    documented there) and commit the results.

docs/html/user_guide.rst

@@ -394,8 +394,8 @@ set like this:
     ignore-installed = true
     no-dependencies = yes
 
-To enable the boolean options ``--no-compile`` and ``--no-cache-dir``, falsy
-values have to be used:
+To enable the boolean options ``--no-compile``, ``--no-warn-script-location``
+and ``--no-cache-dir``, falsy values have to be used:
 
 .. code-block:: ini
 
@@ -404,6 +404,7 @@ values have to be used:
 
     [install]
     no-compile = no
+    no-warn-script-location = false
 
 Appending options like ``--find-links`` can be written on multiple lines:
 

news/31044E84-3F3C-48A8-84B2-6028E21FEBF1.trivial

@@ -0,0 +1,2 @@
+Enforce PEP 508 requirement format in ``pyproject.toml``
+``build-system.requires``.

news/6410.bugfix

@@ -189,7 +189,7 @@ def prepare_release(session):
 def build_release(session):
     version = release.get_version_from_arguments(session.posargs)
     if not version:
-        session.error("Usage: nox -s upload-release -- YY.N[.P]")
+        session.error("Usage: nox -s build-release -- YY.N[.P]")
 
     session.log("# Ensure no files in dist/")
     if release.have_files_in_folder("dist"):
@@ -209,3 +209,36 @@ def build_release(session):
 
     session.log("# Checkout the master branch")
     session.run("git", "checkout", "master", external=True, silent=True)
+
+
+@nox.session(name="upload-release")
+def upload_release(session):
+    version = release.get_version_from_arguments(session.posargs)
+    if not version:
+        session.error("Usage: nox -s upload-release -- YY.N[.P]")
+
+    session.log("# Install dependencies")
+    session.install("twine")
+
+    distribution_files = glob.glob("dist/*")
+    session.log(f"# Distribution files: {distribution_files}")
+
+    # Sanity check: Make sure there's 2 distribution files.
+    count = len(distribution_files)
+    if count != 2:
+        session.error(
+            f"Expected 2 distribution files for upload, got {count}. "
+            f"Remove dist/ and run 'nox -s build-release -- {version}'"
+        )
+    # Sanity check: Make sure the files are correctly named.
+    expected_distribution_files = [
+        f"pip-{version}-py2.py3-none-any.whl",
+        f"pip-{version}.tar.gz",
+    ]
+    if sorted(distribution_files) != sorted(expected_distribution_files):
+        session.error(
+            f"Distribution files do not seem to be for {version} release."
+        )
+
+    session.log("# Upload distributions")
+    session.run("twine", "upload", *distribution_files)

noxfile.py

@@ -10,7 +10,7 @@
 from pip._internal.cli.cmdoptions import make_target_python
 from pip._internal.cli.req_command import RequirementCommand
 from pip._internal.req import RequirementSet
-from pip._internal.req.req_tracker import RequirementTracker
+from pip._internal.req.req_tracker import get_requirement_tracker
 from pip._internal.utils.filesystem import check_path_owner
 from pip._internal.utils.misc import ensure_dir, normalize_path, write_output
 from pip._internal.utils.temp_dir import TempDirectory
@@ -111,7 +111,7 @@ def run(self, options, args):
             )
             options.cache_dir = None
 
-        with RequirementTracker() as req_tracker, TempDirectory(
+        with get_requirement_tracker() as req_tracker, TempDirectory(
             options.build_dir, delete=build_delete, kind="download"
         ) as directory:
 

src/pip/_internal/commands/download.py

@@ -31,7 +31,7 @@
 from pip._internal.locations import distutils_scheme
 from pip._internal.operations.check import check_install_conflicts
 from pip._internal.req import RequirementSet, install_given_reqs
-from pip._internal.req.req_tracker import RequirementTracker
+from pip._internal.req.req_tracker import get_requirement_tracker
 from pip._internal.utils.filesystem import check_path_owner, test_writable_dir
 from pip._internal.utils.misc import (
     ensure_dir,
@@ -343,7 +343,7 @@ def run(self, options, args):
             )
             options.cache_dir = None
 
-        with RequirementTracker() as req_tracker, TempDirectory(
+        with get_requirement_tracker() as req_tracker, TempDirectory(
             options.build_dir, delete=build_delete, kind="install"
         ) as directory:
             requirement_set = RequirementSet(

src/pip/_internal/commands/install.py

@@ -13,7 +13,7 @@
 from pip._internal.cli.req_command import RequirementCommand
 from pip._internal.exceptions import CommandError, PreviousBuildDirError
 from pip._internal.req import RequirementSet
-from pip._internal.req.req_tracker import RequirementTracker
+from pip._internal.req.req_tracker import get_requirement_tracker
 from pip._internal.utils.temp_dir import TempDirectory
 from pip._internal.utils.typing import MYPY_CHECK_RUNNING
 from pip._internal.wheel_builder import WheelBuilder
@@ -157,7 +157,7 @@ def run(self, options, args):
         build_delete = (not (options.no_clean or options.build_dir))
         wheel_cache = WheelCache(options.cache_dir, options.format_control)
 
-        with RequirementTracker() as req_tracker, TempDirectory(
+        with get_requirement_tracker() as req_tracker, TempDirectory(
             options.build_dir, delete=build_delete, kind="wheel"
         ) as directory:
 

src/pip/_internal/commands/wheel.py

@@ -27,8 +27,8 @@
 
 if MYPY_CHECK_RUNNING:
     from typing import (
-        Callable, Dict, Iterable, List, MutableMapping, Optional, Sequence,
-        Tuple, Union,
+        Callable, Iterable, List, MutableMapping, Optional, Sequence, Tuple,
+        Union,
     )
     import xml.etree.ElementTree
 
@@ -435,29 +435,36 @@ def sort_path(path):
 class CollectedLinks(object):
 
     """
-    Encapsulates all the Link objects collected by a call to
-    LinkCollector.collect_links(), stored separately as--
+    Encapsulates the return value of a call to LinkCollector.collect_links().
+
+    The return value includes both URLs to project pages containing package
+    links, as well as individual package Link objects collected from other
+    sources.
+
+    This info is stored separately as:
 
     (1) links from the configured file locations,
     (2) links from the configured find_links, and
-    (3) a dict mapping HTML page url to links from that page.
+    (3) urls to HTML project pages, as described by the PEP 503 simple
+        repository API.
     """
 
     def __init__(
         self,
-        files,       # type: List[Link]
-        find_links,  # type: List[Link]
-        pages,       # type: Dict[str, List[Link]]
+        files,         # type: List[Link]
+        find_links,    # type: List[Link]
+        project_urls,  # type: List[Link]
     ):
         # type: (...) -> None
         """
         :param files: Links from file locations.
         :param find_links: Links from find_links.
-        :param pages: A dict mapping HTML page url to links from that page.
+        :param project_urls: URLs to HTML project pages, as described by
+            the PEP 503 simple repository API.
         """
         self.files = files
         self.find_links = find_links
-        self.pages = pages
+        self.project_urls = project_urls
 
 
 class LinkCollector(object):
@@ -483,18 +490,12 @@ def find_links(self):
         # type: () -> List[str]
         return self.search_scope.find_links
 
-    def _get_pages(self, locations):
-        # type: (Iterable[Link]) -> Iterable[HTMLPage]
+    def fetch_page(self, location):
+        # type: (Link) -> Optional[HTMLPage]
         """
-        Yields (page, page_url) from the given locations, skipping
-        locations that have errors.
+        Fetch an HTML page containing package links.
         """
-        for location in locations:
-            page = _get_html_page(location, session=self.session)
-            if page is None:
-                continue
-
-            yield page
+        return _get_html_page(location, session=self.session)
 
     def collect_links(self, project_name):
         # type: (str) -> CollectedLinks
@@ -537,12 +538,8 @@ def collect_links(self, project_name):
             lines.append('* {}'.format(link))
         logger.debug('\n'.join(lines))
 
-        pages_links = {}
-        for page in self._get_pages(url_locations):
-            pages_links[page.url] = list(parse_links(page))
-
         return CollectedLinks(
             files=file_links,
             find_links=find_link_links,
-            pages=pages_links,
+            project_urls=url_locations,
         )

src/pip/_internal/index/collector.py

@@ -19,6 +19,7 @@
     InvalidWheelFilename,
     UnsupportedWheel,
 )
+from pip._internal.index.collector import parse_links
 from pip._internal.models.candidate import InstallationCandidate
 from pip._internal.models.format_control import FormatControl
 from pip._internal.models.link import Link
@@ -778,6 +779,25 @@ def evaluate_links(self, link_evaluator, links):
 
         return candidates
 
+    def process_project_url(self, project_url, link_evaluator):
+        # type: (Link, LinkEvaluator) -> List[InstallationCandidate]
+        logger.debug(
+            'Fetching project page and analyzing links: %s', project_url,
+        )
+        html_page = self._link_collector.fetch_page(project_url)
+        if html_page is None:
+            return []
+
+        page_links = list(parse_links(html_page))
+
+        with indent_log():
+            package_links = self.evaluate_links(
+                link_evaluator,
+                links=page_links,
+            )
+
+        return package_links
+
     def find_all_candidates(self, project_name):
         # type: (str) -> List[InstallationCandidate]
         """Find all available InstallationCandidate for project_name
@@ -798,14 +818,11 @@ def find_all_candidates(self, project_name):
         )
 
         page_versions = []
-        for page_url, page_links in collected_links.pages.items():
-            logger.debug('Analyzing links from page %s', page_url)
-            with indent_log():
-                new_versions = self.evaluate_links(
-                    link_evaluator,
-                    links=page_links,
-                )
-                page_versions.extend(new_versions)
+        for project_url in collected_links.project_urls:
+            package_links = self.process_project_url(
+                project_url, link_evaluator=link_evaluator,
+            )
+            page_versions.extend(package_links)
 
         file_versions = self.evaluate_links(
             link_evaluator,

src/pip/_internal/index/package_finder.py

@@ -5,6 +5,7 @@
 import sys
 
 from pip._vendor import pytoml, six
+from pip._vendor.packaging.requirements import InvalidRequirement, Requirement
 
 from pip._internal.exceptions import InstallationError
 from pip._internal.utils.typing import MYPY_CHECK_RUNNING
@@ -150,6 +151,21 @@ def load_pyproject_toml(
             reason="'build-system.requires' is not a list of strings.",
         ))
 
+    # Each requirement must be valid as per PEP 508
+    for requirement in requires:
+        try:
+            Requirement(requirement)
+        except InvalidRequirement:
+            raise InstallationError(
+                error_template.format(
+                    package=req_name,
+                    reason=(
+                        "'build-system.requires' contains an invalid "
+                        "requirement: {!r}".format(requirement)
+                    ),
+                )
+            )
+
     backend = build_system.get("build-backend")
     check = []  # type: List[str]
     if backend is None: