Merge pull request #10771 from sbidoul/install-report-sbi

Installation/resolution report (aka pip install --dry-run --report)

Author: sbidoul

docs/html/cli/pip_install.rst

@@ -79,6 +79,18 @@ for an exception regarding pre-release versions). Where more than one source of
 the chosen version is available, it is assumed that any source is acceptable
 (as otherwise the versions would differ).
 
+Obtaining information about what was installed
+----------------------------------------------
+
+The install command has a ``--report`` option that will generate a JSON report of what
+pip has installed. In combination with the ``--dry-run`` and ``--ignore-installed`` it
+can be used to *resolve* a set of requirements without actually installing them.
+
+The report can be written to a file, or to standard output (using ``--report -`` in
+combination with ``--quiet``).
+
+The format of the JSON report is described in :doc:`../reference/installation-report`.
+
 Installation Order
 ------------------
 

docs/html/reference/index.md

@@ -9,4 +9,5 @@ interoperability standards that pip utilises/implements.
 build-system/index
 requirement-specifiers
 requirements-file-format
+installation-report
 ```

docs/html/reference/installation-report.md

@@ -0,0 +1,176 @@
+# Installation Report
+
+The `--report` option of the pip install command produces a detailed JSON report of what
+it did install (or what it would have installed, if used with the `--dry-run` option).
+
+## Specification
+
+The report is a JSON object with the following properties:
+
+- `version`: the string `0`, denoting that the installation report is an experimental
+  feature. This value will change to `1`, when the feature is deemed stable after
+  gathering user feedback (likely in pip 22.3 or 23.0). Backward incompatible changes
+  may be introduced in version `1` without notice. After that, it will change only if
+  and when backward incompatible changes are introduced, such as removing mandatory
+  fields or changing the semantics or data type of existing fields. The introduction of
+  backward incompatible changes will follow the usual pip processes such as the
+  deprecation cycle or feature flags. Tools must check this field to ensure they support
+  the corresponding version.
+
+- `pip_version`: a string with the version of pip used to produce the report.
+
+- `install`: an array of [InstallationReportItem](InstallationReportItem) representing
+  the distribution packages (to be) installed.
+
+- `environment`: an object describing the environment where the installation report was
+  generated. See [PEP 508 environment
+  markers](https://peps.python.org/pep-0508/#environment-markers) for more information.
+  Values have a string type.
+
+(InstallationReportItem)=
+
+An `InstallationReportItem` is an object describing a (to be) installed distribution
+package with the following properties:
+
+- `metadata`: the metadata of the distribution, converted to a JSON object according to
+  the [PEP 566
+  transformation](https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata).
+
+- `is_direct`: `true` if the requirement was provided as, or constrained to, a direct
+  URL reference. `false` if the requirements was provided as a name and version
+  specifier.
+
+- `download_info`: Information about the artifact (to be) downloaded for installation,
+  using the [direct
+  URL](https://packaging.python.org/en/latest/specifications/direct-url/) data
+  structure. When `is_direct` is `true`, this field is the same as the `direct_url.json`
+  metadata, otherwise it represents the URL of the artifact obtained from the index or
+  `--find-links`.
+
+  ```{note}
+  For source archives, `download_info.archive_info.hash` may
+  be absent when the requirement was installed from the wheel cache
+  and the cache entry was populated by an older pip version that did not
+  record the origin URL of the downloaded artifact.
+  ```
+
+- `requested`: `true` if the requirement was explicitly provided by the user, either
+  directely via a command line argument or indirectly via a requirements file. `false`
+  if the requirement was installed as a dependency of another requirement.
+
+- `requested_extras`: extras requested by the user. This field is only present when the
+  `requested` field is true.
+
+## Example
+
+The following command:
+
+```console
+pip install \
+  --ignore-installed --dry-run --quiet \
+  --report - \
+  "pydantic>=1.9" git+https://github.com/pypa/packaging@main
+```
+
+will produce an output similar to this (metadata abriged for brevity):
+
+```json
+{
+  "version": "0",
+  "pip_version": "22.2",
+  "install": [
+    {
+      "download_info": {
+        "url": "https://files.pythonhosted.org/packages/a4/0c/fbaa7319dcb5eecd3484686eb5a5c5702a6445adb566f01aee6de3369bc4/pydantic-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
+        "archive_info": {
+          "hash": "sha256=18f3e912f9ad1bdec27fb06b8198a2ccc32f201e24174cec1b3424dda605a310"
+        }
+      },
+      "is_direct": false,
+      "requested": true,
+      "metadata": {
+        "name": "pydantic",
+        "version": "1.9.1",
+        "requires_dist": [
+          "typing-extensions (>=3.7.4.3)",
+          "dataclasses (>=0.6) ; python_version < \"3.7\"",
+          "python-dotenv (>=0.10.4) ; extra == 'dotenv'",
+          "email-validator (>=1.0.3) ; extra == 'email'"
+        ],
+        "requires_python": ">=3.6.1",
+        "provides_extra": [
+          "dotenv",
+          "email"
+        ]
+      }
+    },
+    {
+      "download_info": {
+        "url": "https://github.com/pypa/packaging",
+        "vcs_info": {
+          "vcs": "git",
+          "requested_revision": "main",
+          "commit_id": "4f42225e91a0be634625c09e84dd29ea82b85e27"
+        }
+      },
+      "is_direct": true,
+      "requested": true,
+      "metadata": {
+        "name": "packaging",
+        "version": "21.4.dev0",
+        "requires_dist": [
+          "pyparsing (!=3.0.5,>=2.0.2)"
+        ],
+        "requires_python": ">=3.7"
+      }
+    },
+    {
+      "download_info": {
+        "url": "https://files.pythonhosted.org/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl",
+        "archive_info": {
+          "hash": "sha256=5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
+        }
+      },
+      "is_direct": false,
+      "requested": false,
+      "metadata": {
+        "name": "pyparsing",
+        "version": "3.0.9",
+        "requires_dist": [
+          "railroad-diagrams ; extra == \"diagrams\"",
+          "jinja2 ; extra == \"diagrams\""
+        ],
+        "requires_python": ">=3.6.8"
+      }
+    },
+    {
+      "download_info": {
+        "url": "https://files.pythonhosted.org/packages/75/e1/932e06004039dd670c9d5e1df0cd606bf46e29a28e65d5bb28e894ea29c9/typing_extensions-4.2.0-py3-none-any.whl",
+        "archive_info": {
+          "hash": "sha256=6657594ee297170d19f67d55c05852a874e7eb634f4f753dbd667855e07c1708"
+        }
+      },
+      "is_direct": false,
+      "requested": false,
+      "metadata": {
+        "name": "typing_extensions",
+        "version": "4.2.0",
+        "requires_python": ">=3.7"
+      }
+    }
+  ],
+  "environment": {
+    "implementation_name": "cpython",
+    "implementation_version": "3.10.5",
+    "os_name": "posix",
+    "platform_machine": "x86_64",
+    "platform_release": "5.13-generic",
+    "platform_system": "Linux",
+    "platform_version": "...",
+    "python_full_version": "3.10.5",
+    "platform_python_implementation": "CPython",
+    "python_version": "3.10",
+    "sys_platform": "linux"
+  }
+}
+```

news/53.feature.rst

@@ -0,0 +1,3 @@
+Add an experimental ``--report`` option to the install command to generate a JSON report
+of what was installed. In combination with ``--dry-run`` and ``--ignore-installed`` it
+can be used to resolve the requirements.

src/pip/_internal/commands/install.py

@@ -1,4 +1,5 @@
 import errno
+import json
 import operator
 import os
 import shutil
@@ -7,6 +8,7 @@
 from typing import Iterable, List, Optional
 
 from pip._vendor.packaging.utils import canonicalize_name
+from pip._vendor.rich import print_json
 
 from pip._internal.cache import WheelCache
 from pip._internal.cli import cmdoptions
@@ -21,6 +23,7 @@
 from pip._internal.locations import get_scheme
 from pip._internal.metadata import get_environment
 from pip._internal.models.format_control import FormatControl
+from pip._internal.models.installation_report import InstallationReport
 from pip._internal.operations.build.build_tracker import get_build_tracker
 from pip._internal.operations.check import ConflictDetails, check_install_conflicts
 from pip._internal.req import install_given_reqs
@@ -250,6 +253,20 @@ def add_options(self) -> None:
         self.parser.insert_option_group(0, index_opts)
         self.parser.insert_option_group(0, self.cmd_opts)
 
+        self.cmd_opts.add_option(
+            "--report",
+            dest="json_report_file",
+            metavar="file",
+            default=None,
+            help=(
+                "Generate a JSON file describing what pip did to install "
+                "the provided requirements. "
+                "Can be used in combination with --dry-run and --ignore-installed "
+                "to 'resolve' the requirements. "
+                "When - is used as file name it writes to stdout."
+            ),
+        )
+
     @with_cleanup
     def run(self, options: Values, args: List[str]) -> int:
         if options.use_user_site and options.target_dir is not None:
@@ -353,6 +370,20 @@ def run(self, options: Values, args: List[str]) -> int:
                 reqs, check_supported_wheels=not options.target_dir
             )
 
+            if options.json_report_file:
+                logger.warning(
+                    "--report is currently an experimental option. "
+                    "The output format may change in a future release "
+                    "without prior warning."
+                )
+
+                report = InstallationReport(requirement_set.requirements_to_install)
+                if options.json_report_file == "-":
+                    print_json(data=report.to_dict())
+                else:
+                    with open(options.json_report_file, "w", encoding="utf-8") as f:
+                        json.dump(report.to_dict(), f, indent=2, ensure_ascii=False)
+
             if options.dry_run:
                 would_install_items = sorted(
                     (r.metadata["name"], r.metadata["version"])

src/pip/_internal/models/installation_report.py

@@ -0,0 +1,53 @@
+from typing import Any, Dict, Sequence
+
+from pip._vendor.packaging.markers import default_environment
+
+from pip import __version__
+from pip._internal.req.req_install import InstallRequirement
+
+
+class InstallationReport:
+    def __init__(self, install_requirements: Sequence[InstallRequirement]):
+        self._install_requirements = install_requirements
+
+    @classmethod
+    def _install_req_to_dict(cls, ireq: InstallRequirement) -> Dict[str, Any]:
+        assert ireq.download_info, f"No download_info for {ireq}"
+        res = {
+            # PEP 610 json for the download URL. download_info.archive_info.hash may
+            # be absent when the requirement was installed from the wheel cache
+            # and the cache entry was populated by an older pip version that did not
+            # record origin.json.
+            "download_info": ireq.download_info.to_dict(),
+            # is_direct is true if the requirement was a direct URL reference (which
+            # includes editable requirements), and false if the requirement was
+            # downloaded from a PEP 503 index or --find-links.
+            "is_direct": bool(ireq.original_link),
+            # requested is true if the requirement was specified by the user (aka
+            # top level requirement), and false if it was installed as a dependency of a
+            # requirement. https://peps.python.org/pep-0376/#requested
+            "requested": ireq.user_supplied,
+            # PEP 566 json encoding for metadata
+            # https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata
+            "metadata": ireq.get_dist().metadata_dict,
+        }
+        if ireq.user_supplied and ireq.extras:
+            # For top level requirements, the list of requested extras, if any.
+            res["requested_extras"] = list(sorted(ireq.extras))
+        return res
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "version": "0",
+            "pip_version": __version__,
+            "install": [
+                self._install_req_to_dict(ireq) for ireq in self._install_requirements
+            ],
+            # https://peps.python.org/pep-0508/#environment-markers
+            # TODO: currently, the resolver uses the default environment to evaluate
+            # environment markers, so that is what we report here. In the future, it
+            # should also take into account options such as --python-version or
+            # --platform, perhaps under the form of an environment_override field?
+            # https://github.com/pypa/pip/issues/11198
+            "environment": default_environment(),
+        }