Report per-object progress through WithProgress

[imjasonh]

Based on conversation in #967 (comment)

It'd be nice to have WithProgress expose per-blob and per-manifest upload progress, to diagnose issues and provide a nice UX like you get with docker pull.

Off the top of my head we could add fields to v1.Update to support this:

type Update struct {
  // existing fields
  Total    int64
  Complete int64
  Error    error

  // new fields
  ObjectName string
  ObjectTotal, ObjectComplete int64
}

(naming TBD)

Then when we read and upload a blob using progressReader, have each progressReader also take the name of the object it's reading, and count per-reader (blob) progress.

A caller that's only interested in the full overall progress, they should just be able to ignore the new per-object fields and consume Total/Complete as normal.

Manifests are uploaded in one request, so those won't get fine-grained progress updates, but they're small anyway.

cc @deitch

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[deitch]

Hi @imjasonh picking this up again after a long time.

This originally was about pushing to a remote, but I think it also is relevant to reading from a remote. For a small image (10MB) pulling to the cloud, it might not make sense, but even 10MB to an edge device, let alone a VM disk image (hundreds of MB or more), or an LLM model, these can be really big.

For example (skipping error handling and such):

	desc, err := remote.Get(ref, options...)
	p, err := layout.FromPath(savePath)
	ii, err := desc.ImageIndex()
	err = p.AppendIndex(ii)

If the manifest or layers take a while, we would want to see progress.

I thought about using:

	c := make(chan<- v1.Update)
	options = append(options, remote.WithProgress(c))

But that doesn't work for a few reasons:

• I don't think that the pulling part ever uses progress, as evidenced by remote.Puller.get and remote.Puller.Layer
• progress as defined as v1.Update would struggle with the multiple pulls, as it does not indicate what it is for:

type Update struct {
	Total    [int64](https://pkg.go.dev/builtin#int64)
	Complete [int64](https://pkg.go.dev/builtin#int64)
	Error    [error](https://pkg.go.dev/builtin#error)
}

It would need either to include the digest of the blob being pulled in the Update, so the receiver on the channel can demultiplex what it receives, or create multiple channels.

As always, I am happy to create the necessary PRs for this, as long as I know they will be accepted and the work won't be for naught.

I could use some direction as well.

I think that:

• pass progress option (chan) from remote.Puller to instances of remote.fetcher, when created
• extend v1.Update to include the blob digest (maybe media-type as well? Maybe the desc?)
• send updates to the channel as pulling. For getting manifests, that appears to be in fetcher.get(), for layers in remote.remoteImageLayer.Compressed() (and probably somewhere else? interesting that one is in fetcher and one is not), for config file and others, I am not sure where it is.

Thoughts?

[deitch]

Also, how can I remove the frozen lifecycle from it?