Start Publishing Staging Images to Artifact Registry

[upodroid]

Part of: #1343

Notes: https://gist.github.com/upodroid/a33723a7e1abc5e9c6fabc6b07e7aac0

When images are built after a PR is merged, they need to be pushed to Artifact Registry(AR).

Unlike Google Container Registry(GCR), you don't need a separate GCP project per staging project. The permission boundary is at the repository level instead of the project. This allows multiple registries to be created per project. Therefore, we will create a new project, start publishing images to it and delete the other projects after the transition.

We will need to do the following:
- create a new GCP project (k8s-artifacts-staging)
- create a docker image repository per project
- modify the jobs to push images to prow.
- Provision the infra with terraform. There is a separate issue open to deploy changes to terraform via Prow on merges.

Prod changes need to be done via the shell scripts which makes deploying staging via terraform kind of pointless.

Open questions:

• Do we need to backfill the staging repository via gcrane?

@puerco

/area artifacts
/priority important-soon
/area release-eng

[upodroid]

After #3968 is merged, we need to do a few things:

• repos that use ko to build images(k-sigs/bom,etc), modify KO_DOCKER_REPO variable to us-docker.pkg.dev/k8s-staging-bom/images/bom as an example
• Modify image-builder to tweak the cloudbuild.yaml file

Bit unsure about other things.

[upodroid]

Spoke about this at the sig-testing meeting. Will push it out to the future when GCR deprecation is actually announced by Google.

May want to standardize on Aritfact Registry eventually, but would require tracking down a number of staging registries and changing team push processes (with unclear payoff right now)

/priority backlog
/sig testing

[ameukam]

/milestone v1.26

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone v1.27
/priority important-longterm

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[ameukam]

/remove-lifecycle rotten

[ameukam]

/milestone v1.33

[BenTheElder]

(Some projects are on AR, but not all, thanks to @upodroid)

[ameukam]

We have exceptions for migration:

• https://github.com/kubernetes/k8s.io/blob/main/infra/gcp/terraform/k8s-staging-images/registries.tf#L35-L41

[upodroid]

Quick update, I'm on track to finish the ACTIVE projects by the deadline by Google(18th of March)

Outstanding projects:

# some of these have been moved to k8s-staging-images AR instances such as #7824
 mahamed  Mahameds-MacBook-Pro  ~  1  USAGE  $  gcloud container images list-gcr-usage     --folder=713040427754 --filter="usage=ACTIVE"
---
repository: gcr.io/k8s-staging-lws
usage: ACTIVE
---
repository: gcr.io/k8s-staging-jobset
usage: ACTIVE
---
repository: gcr.io/k8s-staging-provider-os
usage: ACTIVE
---
repository: gcr.io/k8s-staging-tejolote
usage: ACTIVE
---
repository: gcr.io/k8s-staging-zeitgeist
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kmm
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kwok
usage: ACTIVE
---
repository: gcr.io/k8s-staging-win-op-rdnss
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kueue
usage: ACTIVE
---
repository: gcr.io/k8s-staging-gmsa-webhook
usage: ACTIVE
---
repository: gcr.io/k8s-staging-sched-simulator
usage: ACTIVE
---
repository: gcr.io/k8s-staging-prometheus-adapter
usage: ACTIVE
---
repository: gcr.io/k8s-staging-gateway-api
usage: ACTIVE
---
repository: gcr.io/k8s-staging-releng-test
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kustomize
usage: ACTIVE
---
repository: gcr.io/k8s-staging-networking
usage: ACTIVE
---
repository: gcr.io/k8s-staging-sp-operator
usage: ACTIVE
---
repository: gcr.io/k8s-staging-sig-docs
usage: ACTIVE
---
repository: gcr.io/k8s-staging-scheduler-plugins
usage: ACTIVE
---
repository: gcr.io/k8s-staging-git-sync
usage: ACTIVE
---
repository: gcr.io/k8s-staging-sig-storage
usage: ACTIVE
---
repository: gcr.io/k8s-staging-ingressconformance
usage: ACTIVE
---
repository: gcr.io/k8s-staging-ingress-nginx
usage: ACTIVE
---
repository: gcr.io/k8s-staging-provider-aws
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kube-state-metrics
usage: ACTIVE
---
repository: gcr.io/k8s-staging-slack-infra
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kubernetes
usage: ACTIVE
---
repository: gcr.io/k8s-staging-scl-image-builder
usage: ACTIVE
---
repository: gcr.io/k8s-staging-nfd
usage: ACTIVE
---
repository: gcr.io/k8s-staging-npd
usage: ACTIVE
---
repository: gcr.io/k8s-staging-external-dns
usage: ACTIVE
---
repository: gcr.io/k8s-staging-provider-azure
usage: ACTIVE
---
repository: gcr.io/k8s-staging-kas-network-proxy
usage: ACTIVE
---
repository: gcr.io/k8s-staging-multitenancy
usage: ACTIVE
---
repository: gcr.io/k8s-staging-publishing-bot
usage: ACTIVE

Successfully Migrated Projects Projects:

 mahamed  Mahameds-MacBook-Pro  ~  1  $  gcloud container images list-gcr-usage     --folder=713040427754 --filter="usage=REDIRECTED"
---
repository: gcr.io/k8s-staging-cloud-pv-vsphere
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-ipam-ic
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cloud-pv-labeler
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-helm
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-operator
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-cloudstack
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-bom
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cloud-provider-ibm
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-ibmcloud
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-nested
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-test-infra
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cri-tools
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-ci-images
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-addon-manager
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-addons
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cloud-provider-gcp
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-boskos
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-infra-tools
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cpa
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-etcd
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-csi-secrets-store
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-dns
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-do
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-autoscaling
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-vsphere
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-apisnoop
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-e2e-test-images
usage: REDIRECTED
---
repository: asia.gcr.io/k8s-staging-cip-test
usage: REDIRECTED
---
repository: eu.gcr.io/k8s-staging-cip-test
usage: REDIRECTED
---
repository: us.gcr.io/k8s-staging-cip-test
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-azure
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-gcp
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-docker
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-openstack
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-artifact-promoter
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-capi-kubeadm
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-descheduler
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-build-image
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api-aws
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cip-test
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-coredns
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-csi
usage: REDIRECTED
---
repository: gcr.io/k8s-staging-cluster-api
usage: REDIRECTED

Projects with inactive GCR usage. They will be migrated in late March after the ACTIVE projects are done.

 mahamed  Mahameds-MacBook-Pro  ~  1  $  gcloud container images list-gcr-usage     --folder=713040427754 --filter="usage=INACTIVE"
---
repository: gcr.io/k8s-staging-wasm-scheduler
usage: INACTIVE
---
repository: gcr.io/k8s-staging-kubebuilder
usage: INACTIVE
---
repository: gcr.io/k8s-staging-win-svc-proxy
usage: INACTIVE
---
repository: gcr.io/k8s-staging-tg-exporter
usage: INACTIVE
---
repository: gcr.io/k8s-staging-krm-functions
usage: INACTIVE
---
repository: gcr.io/k8s-staging-sig-auth
usage: INACTIVE
---
repository: gcr.io/k8s-staging-kubetest2
usage: INACTIVE
---
repository: gcr.io/k8s-staging-provider-openstack
usage: INACTIVE
---
repository: gcr.io/k8s-staging-experimental
usage: INACTIVE
---
repository: gcr.io/k8s-staging-examples
usage: INACTIVE
---
repository: gcr.io/k8s-staging-storage-migrator
usage: INACTIVE
---
repository: gcr.io/k8s-staging-etcdadm
usage: INACTIVE
---
repository: gcr.io/k8s-staging-k8s-gsm-tools
usage: INACTIVE
---
repository: gcr.io/k8s-staging-kubeadm
usage: INACTIVE

[ameukam]

@upodroid Are we not done with this ?

[thockin]

I can't tell if git-sync was done or not -- if so, is there anything I need to DO?

[BenTheElder]

@thockin TLDR: yes git-sync was moved and you don't need to do anything.

x-ref kubernetes/git-sync#940 (comment)

---

Confirming:

x-ref etcd-io/etcd#15199 (comment)

# check for legacy GCR, which used storage.googleapis.com
crane pull --verbose gcr.io/k8s-staging-git-sync/git-sync:v3.6.9 /dev/null 2>&1 | grep storage
# nothing

# check for artifact registry
$ crane pull --verbose gcr.io/k8s-staging-git-sync/git-sync:v3.6.9 /dev/null 2>&1 | grep /artifacts-downloads | head -n1
Location: /artifacts-downloads/namespaces/k8s-staging-git-sync/repositories/gcr.io/downloads/[snipped super long hash url]

You, as an owner of it, can also browse to gcr.io/k8s-staging-git-sync and see that it redirects you to console.cloud.google.com/artifacts/docker/k8s-staging-git-sync/us/gcr.io

You don't need to do anything, there's some loose plans to migrate to AR domains, but it's not essential, for now it is migrated via https://cloud.google.com/artifact-registry/docs/transition/auto-migrate-gcr-ar and you don't need to do anything.

[BenTheElder]

I think we can close this.
We can separately track attempts to shift to AR domains directly.