PEP 751 experimental pip lock command

[sbidoul]

This is a rough PoC for a pip lock command that writes PEP 751 compliant toml to stdout.

Use with:

• pip --quiet lock -o - . to lock the local project to stdout.
• pip lock -e . to lock the local project in editable mode to pylock.toml

I wrote this mainly as a way to better understand the PEP.

A few comments:

• This command works with the same logic as pip install --dry-run --ignore-installed, with mostly the same options.
• I removed the platform/abi/implementation/python-version options due to their limitations (Various package-index filtering flags do not affect the environment markers #11664). We could keep them with the same caveat as for the other commands...
• ...but it is unclear to me if pip in its current state could output something useful for environments, requires-python, packages.marker.
• This command begs for a --only-deps option (Add --only-deps (and --only-build-deps) option(s) #11440)
• The tables are currently not inlined which makes the output somewhat hard to grok.

[potiuk]

Nice. I would love to help with making it happen (of course it depends on PEP 751 to be approved). We are currently doing our own custom and pretty "poor" implementation of locks with cosntraints in Airlfow and having a "standard" way of reproducible installs would be a great thing.

Not sure what is the status now of the PEP and whether it has a chance to be approved soon, but that looks really great that we are trying to standardise it now.

[notatallshaw]

I appreciate this is draft, but some small early feedback.

Firstly, I am advocating for an "output file" option. Writing pip's output to a file in other commands has several footguns, e.g knowing the correct flags to turn off non-relevant output, determining if pip hit errors or warnings when you do, and writing non-UTF-8 files when you direct standard out from powershell on Windows.

pip-compile (and uv pip compile) solve this by having an output file option which is considered best practice to use. That said, I don't know the history of why other commands don't offer this, perhaps it's non-trivial.

I removed the platform/abi/implementation/python-version options due to their limitations

I agree, I think pip should start as limited as possible around known difficult issues, and expand from there. This will give the chance for the lock API to evolve, if able.

[pfmoore]

Cool. I did some early PoC work on the PEP, but things have evolved drastically since then, and I never got back to it. But one of my goals was always for the pip installation report format to be easily convertible into a lockfile. Another thing I'd like to see pip implement (I may well get round to this myself, in due course, it's not a request for someone else to pick it up!) is pip freeze --format=lockfile. I'd hope that would be relatively easy to implement.

I'd very definitely aim pip at solely producing "environment reproduction" lockfiles, and not cross-platform ones. To that end, I think that disallowing the --platform etc., arguments¹ is the right call.

I don't have a strong opinion on the question of an --output-file option, but I think that defaulting to writing to stdout is a reasonable place to start. With an output file option, we'd have to consider validating the provided filename to match the standard-required form pylock[.xxx].toml, which IMO is a distraction from the main exercise of ensuring that we produce a correct lockfile.

Not sure what is the status now of the PEP and whether it has a chance to be approved soon

The PEP is in its final stages before being submitted for pronouncement. At this point, it's extremely unlikely that any major changes will occur. Speaking as the PEP-delegate, I can say that I've been heavily involved in the discussions on the PEP, and I'm confident that it has a good chance of being accepted².

Footnotes

1. Or, as you say, putting them under a "use at your own risk" warning. ↩

2. I'm not giving anything away here - I've always been supportive of the idea of a lockfile standard, but the problem has been getting community consensus on what it should look like. At this point, we're the closest we've ever been to consensus. ↩

[sbidoul]

pip freeze --format=lockfile

I think a pre-requisite for that is PEP 710

[notatallshaw]

I don't have a strong opinion on the question of an --output-file option, but I think that defaulting to writing to stdout is a reasonable place to start. With an output file option, we'd have to consider validating the provided filename to match the standard-required form pylock[.xxx].toml, which IMO is a distraction from the main exercise of ensuring that we produce a correct lockfile.

I'm fine with this being in a follow up PR if there are non-trivial questions to answer.

Though looking now at the PEP I do think a pedantic reading of it would be that the data should only ever be written as a file and not outputted as stdout as that would technically violate those same file name specifications 😉. (If not made clear by emoji I think that's terrible and that stdout should absolutely be an option at least).

[pfmoore]

🙂 I wasn't trying to be pedantic, just noting the likelihood that someone would ask the question about validating the filename. Getting the encoding correct by using the --output-file option is a much more important point, though.

[sbidoul]

I choose not to emit version for direct URL packages, as the version could by dynamic. This is to avoid problems with projects using setuptools-scm, where the version possibly changes with each commit and committing the lock file would change the version.

Also, I choose to emit a relative path for directory packages, so the common use case (namely pip lock .) produces something that is portable.

[sbidoul]

Now that PEP 751 is accepted, let's see if we want to move forward with this pip lock command.

A few questions I have:

• Is this CLI good as it is now ?
• Do we want to emit environments ? The only thing that can be done easily is to emit the full current environment, but that is of course way too restrictive in most cases if installers would refuse to install if the installation environment does not match the lock environment. I'm enclined to leave it out in a first iteration.
• The output format could be optimized by inlining some tables. Do we want to improve that ? tomli-w does not support that. tomlkit could be an option but it is heavier and harder to use leading to less readable code. Is there another option? We could also postpone inlining and see if this is actually a problem in practice.

[pfmoore]

+1 from me. I think the CLI is good (we can make changes later if needed). I agree, let's not emit environments for now at least.

Let's not worry about inlining for now. Layout is an area where there are a number of possible options, and I imagine tools will ultimately converge on an approach that provides good readability/auditability and easy generation. It's possible that someone could even produce a dedicated lockfile writer module. I'd rather we stuck with something simple, and with a smaller new library to vendor, until we have a better feel for where the community is going here.

[sbidoul]

Good, I'll see if I can add some tests in time for 25.1.

Except for the absence of tests (and docs?), this is ready to review.

[pfmoore]

I've added the 25.1 milestone, but if you don't have the time (we're about 2 weeks away from the release) feel free to reassign to 25.2 or just remove the milestone.

[ichard26]

I haven't reviewed the PR at all, but if we're going to land this feature for pip 25.1, would it be beneficial to mark this new command as experimental like pip index version? As I said, I haven't reviewed the PR so maybe this command is already well-designed, but I'd rather be honest and give ourselves some breathing room as the ecosystem plays with the new standard. Despite the multi-year discussion, I don't think we'll be able to nail the UI side of lockfiles on our first try.

[uranusjr]

+1 to marking this as experiemental.

[webknjaz]

Looks ready.

[sbidoul]

Thanks for the review everyone!