Ensure that all current Python packaging interoperability standards are on packaging.python.org

[pradyunsg]

This is a matter of someone going through https://peps.python.org/topic/packaging/, identifying all accepted and final PEPs, and converting their content into reference material on this documentation.

This would be content that lives under https://packaging.python.org/en/latest/specifications/, although it will also need to be updated to be the canonical reference -- with consistent structure across all these pages as well as no "see PEP XYZ for details" from these.

[pradyunsg]

x-ref: #955
x-ref: #679
x-ref: #764

[pradyunsg]

x-ref: #765

[CAM-Gerlach]

I've begun work on and hope to submit a PR this week with a draft of the migrated specifications defined in PEP 517/518/660 following the outline I proposed in #955 , presuming no one has major objections to the overall approach (though I've given due consideration to such alternatives, it is relatively flexible to adapt if reviewers feel that, e.g. "Declaring a project's build system" should be a subsection of "Build backend interface specification", or that the hooks in the latter should be organized by required/optional rather than build type).

That seems to be the highest priority missing chunk of specs/PEPs, but there are a few other major ones that have existing placeholders:

• Simple repository API in PEP 503, PEP 592, PEP 629 and PEP 658, though hopefully part of the implementation of the JSON API PEP 691 can be moving over the specification of the existing simple API as well as the new JSON API to packaging.python.org.
• Dependency specifiers in PEPs 345, 440, 508 and now 685
• Version specifiers in PEP 345 and PEP 440
  Those are the pages with current stubs and brief summaries on the PyPA specs page.
• The platform compatibility tag material from PEP 425 should be merged in to the existing, currently-supplementary platform compat tag spec

Other potentially relevant packaging PEPs that declare a formal specification and are possible candidates for migration here include:

• The manylinux wheel compatibility standards (PEP 600, at least, and perhaps even PEP 599)
• Trove classifiers as defined in PEP 301 (PEP referenced in Core Metadata and Project [Source] Metadata); could incorporate any remaining still-relevant content into the Classifier section of Core Metadata
• Virtual environments spec defined in PEP 405; would get its own spec document named such that could basically be a straight copy/paste of the PEP with any required minor updates and editorial tweaks.

There are a handful of other accepted/implemented packaging PEPs, but they are either obsolete/regarding past actions, are specific to PyPI and/or the standard library or are otherwise not relevant to external tools, and/or don't define a normative, living specification as opposed to justifying a specific set of actions to take (e.g. removing support for old distribution formats on PyPI, deprecating distutils, etc).

[CAM-Gerlach]

Somewhat related: I've added/updated admonitions to all the existing migrated PEPs clarifying their status and directing users to the corresponding PyPA specifications in python/peps#2690 , as well as updated the status and replacements/etc. of a few older PEPs. In the process, I discovered a couple more PEPs that should probably be migrated, which I added to the above list.

[CAM-Gerlach]

I'm deep into migrating PEP 517, PEP 518 and PEP 660 on #955 into relevant specs (I just didn't finish before SciPy hit and then I got COVID there, despite taking careful precautions), but I'm back on it now and hope to have an initial PR by early next week, hopefully sooner.

Meanwhile, as outlined above, there are two major clusters that already have stub PyPA specs linking to PEPs that are likely highest-priority for merging:

• The Simple Repository API, which is built out of a bunch of small to medium sized PEPs that would need to be integrated, which would take some thought and planning, and
• The Version and Dependency specifiers, which are essentially just defined in their single respective standalone PEPs (440 and 508) that the normative sections of which could be basically copied in as-is, aside from syntax or minor editorial tweaks and very minor updates to the latter from PEP 685.

There's also merging in the relevant parts of PEP 425 to the existing addendums in the platform tag spec, and a handful of more marginal things.

If you wanted to pick whichever you or want and get started, I was thinking I could finish up 517/518/660 (which you and the others could review), work through my backlog and update my own 639, as well as being a second pair of eyes on your work, and then depending on how far you've gotten in the meantime and your plans then, I could potentially help with some of whatever is left at that point. Does that sound like a plan? Or do you already have something more specific in mind?

[pradyunsg]

That does sound reasonable to me.

I'll likely pick up the specifiers first, since those seem like the more complicated ones to try and move over. :)

[CAM-Gerlach]

The PR for PEP 517/518/660 is now up!

[jeanas]

A lot of this has been done in the meantime. PEP 517 is pending with #1414 (currently waiting on #1396).

Here's my list of what remains:

• PEP 660 Editable installs for pyproject.toml-based builds — I plan to do this one after the PR for PEP 517 is merged.
• PEP 561 Distributing and packaging type information — I plan to do this as well
• PEP 656 musllinux platform tag — same
• PEP 541 Package index name retention — Should this be in the PUG? The system is on hold right now so I'm not sure we want to advertise it more...

There's also a whole slew of PEPs to refine the simple repo API, but I know very little about this topic so I'm not prepared to do that part of the effort.

Did I miss any PEPs? I just went through the entire index of packaging PEPs (plus the discussion above) and these are the ones I noticed but I might have missed part of older PEPs that haven't been imported.

[pradyunsg]

Should this be in the PUG? The system is on hold right now so I'm not sure we want to advertise it more...

No, although it's less a case that things are on hold and more that things are processed as-and-when there's volunteer capacity to process those tickets. And there's a very small set of people with the admin bit on PyPI who can handle them.

Did I miss any PEPs?

No, I believe you covered them all. Thank you for all the work you've put towards this -- it's sincerely appreciated!

[EpicWink]

Is anyone working on the simple repository API PEPs? I can't find any other issues or merge-requests on the topic. I can put some time into them as they've been on my mind recently.

[jeanas]

#1441 is open for PEP 656 / musllinux.

Is anyone working on the simple repository API PEPs? I can't find any other issues or merge-requests on the topic. I can put some time into them as they've been on my mind recently.

This would be great! No, I don't think anybody's working on that.

[jeanas]

I just noticed one more item: this part of PEP 566 (how to transform core metadata into JSON).

[jeanas]

PEP 561 Distributing and packaging type information — I plan to do this as well

After starting to work on it and getting more familiar with the PEP, I'm not so sure PEP 561 should be on packaging.python.org. It looks like the people involved in it were typing people, not packaging people, and it specifies what users and type checkers should do, not what packaging tools should do.

I think it would be good to have a guide on packaging type information in the PUG, but PEP 561 might fit better in the upcoming typing specification.

[jeanas]

I think it would be good to have a guide on packaging type information in the PUG

In fact, this is covered on https://typing.readthedocs.io/en/latest/source/libraries.html. It looks out of date, but let's not duplicate it...