Switch to a dedicated tool for vendoring pip's dependencies

[pradyunsg]

This PR switches pip's vendoring process, to vendoring -- https://pypi.org/project/vendoring/.

vendoring is an independent CLI tool for generating "vendor" directories (like our pip._vendor directory). vendoring is configurable, can do everything pip's current vendoring setup can, has a separate CI + tests¹ and, overall, should be easier to use/maintain than our invoke task.

The configuration file contains all pip-specific quirks of our vendoring process. The plan is, that the configuration file with remain the same as in this PR (mostly) and further improvements will be made to vendoring, which would make the pip's vendor-dependencies process smoother and easier.

¹ For now, vendoring's codebase does not have any useful automated tests, but that's the same situation as pip's current code for vendoring dependencies.

---

Story time!

My initial motivation for working on our vendoring process was that the fetch-licenses step is ridiculously slow.

I tried improving that step but it was slow going. At one point, I realized I'd spent more time fighting the "debt" in that code (i.e. our vendoring logic) vs making improvements to it. I'd spent a fair bit of time trying to improve the code, and concluded that moving this code out of pip's codebase would likely be the best way forward.

So, I spent a... few... days breaking out pip's vendoring logic, into a dedicated tool. That effort resulted in https://github.com/pradyunsg/vendoring. As things stand, vendoring is at feature parity with pip's code for vendoring dependencies. However, I expect it'll only get better with time -- especially if someone works on it (eg. me) since, IMO, it's significantly easier to work on vs our existing setup.

[chrahunt]

Nice job! 2 cosmetic items, but otherwise this LGTM. I compared our previous implementation against the vendoring codebase and I think that having this separate will make it a lot easier to work with. The fact that our vendoring CI passes is a good enough test for now IMO.

We may want to start doing vendored library updates earlier than just before release, to catch any issues that would need to be fixed in vendoring "blocking" our release, though we can work around pretty much anything with patches if needed.

[xavfernandez]

Nice job, and good idea to extract it in a separate package.
Two questions:

• should it be moved to PyPA ?
• did you try applying it to https://github.com/pypa/setuptools/tree/master/setuptools/_vendor ?

[pradyunsg]

I'm on board for both of those tasks (and will be exploring using vendoring for setuptools' and pytest's codebases today).

These tasks weren't critical for pip's adoption of this tool though, so they didn't get prioritized.

[dstufft]

Just a tad bit of history here, I had originally contemplated writing a tool like this, and the Debian folks (might have been Barry Warsaw?) talked me out of it due to a fear that by making the tool easier to consume/use, more projects would rely on vendoring and then they'd have a harder job packaging python libraries in their distro. The flip side of that, is if the tool ensures that vendoring is done cleanly, then projects that were going to vendor anyways can get a much easier to unvendor set of dependencies.

I don't have a strong opinion one way or another, I just happened to see this and wanted to provide some context on why it was the way it was.