PEP 740 persistence, take 3

[woodruffw]

WIP.

Action plan:

• Make a separate PR for the models + migration, to eliminate the zippered reverts that we've had to do in the past.
  • warehouse: PEP 740 models #16625
• Make a separate PR with an appropriate admin flag for disabling provenance persistence.
  • Add a new flag to disable PEP 740 support. #16645
• Investigate the other action items in Revert PEP 740 persistence #16623
• Perform breakouts
  • requirements: bump sigstore, pypi-attestations #16683
  • PEP 740: add IntegrityService and interface #16684
• Add a static route for the provenance object, now that it's no longer being plumbed from object storage
  • Provenance retrieval route #16778
  • PEP 740: add provenance to simple API #16801

This reverts commit b6cf775.

[woodruffw]

I discussed the architectural challenges here a bit with @ewdurbin, and we came to the conclusion that the current scale/scope and expected volume doesn't warrant the complexity that's coming from the attestations feature needing to interact with object storage.

Given that files can currently only have a single attestation, our current conclusion is that the DB should contain a Provenance model, which in turn bears the individual attestation objects/bundles as members of a JSONB list.

In other words, the current data model of File -> [Attestation] will become File -> Provenance?, where Provenance tracks an interior set of attestation bundles as JSONB.

[ewdurbin]

Hm, I think I came to a slightly different mental model than you.

I imagined immutable Attestation objects remaining as a concrete-db-backed model and Provenance being rendered from all attestations related to a file by joining them up then generating a hash. Bonus: this retains the ability for additional attestations to be added down the line.

[woodruffw]

I imagined immutable Attestation objects remaining as a concrete-db-backed model and Provenance being rendered from all attestations related to a file by joining them up then generating a hash. Bonus: this retains the ability for additional attestations to be added down the line.

I messed around with this a bit, but it has the "lifecycle" problem that our earlier design (which avoided persisting provenance objects to artifact storage at all) had: the provenance object needs access to the uploading request object to persist the appropriate Trusted Publisher metadata + claims. We could generate it outside of that request context, but doing so requires another mechanism to hold onto that request state. That becomes especially annoying with the OIDC claims, since those are a free-structured dict that aren't part of the TP state itself 😅

I'm halfway through rebasing this PR on top of #16625, which will hopefully make it a lot simpler and demonstrate how I think (I hope) we can still do mutability while sticking to only a single concrete DB-backed Provenance model. But if the rebased version isn't what you're thinking, I'm happy to undo and try and push forwards with a "collect the provenance from concrete DB-based attestations" approach 🙂

[DarkaMaul]

FLAG: This test is performed in the IntegrityService but was not done here and adding it highlighted some issues.

[woodruffw]

Superseded by #16801.