Failed to pull image "gcr.io/k8s-minikube/storage-provisioner-arm64:v2"

[moritalous]

Steps to reproduce the issue:

I use Raspberry Pi OS 64bit(ARM64) on Raspberry Pi 4.
From version 1.12.2, the minikube create command pulls tag v2 of storage-provisioner-arm64, but it doesn't exist in gcr.io. (Only v1.8.1 exists)
Therefore, the storage-provisioner pod will not start.

minikube v1.12.1 work fine
minikube v1.12.2 not work
minikube v1.12.3 not work

Full output of failed command:

pi@raspberrypi:~ $ sudo minikube start --vm-driver=none
😄  minikube v1.12.3 on Raspbian 10.5 (arm64)
✨  Using the none driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=4, Memory=7816MB, Disk=29460MB) ...
ℹ️  OS release is Debian GNU/Linux 10 (buster)
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.12 ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /root
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"
pi@raspberrypi:~ $ kubectl get pod -n kube-system 
NAME                                  READY   STATUS         RESTARTS   AGE
coredns-66bff467f8-fflnn              1/1     Running        0          70s
etcd-raspberrypi                      1/1     Running        0          68s
kube-apiserver-raspberrypi            1/1     Running        0          68s
kube-controller-manager-raspberrypi   1/1     Running        0          68s
kube-proxy-kzpvp                      1/1     Running        0          70s
kube-scheduler-raspberrypi            1/1     Running        0          68s
storage-provisioner                   0/1     ErrImagePull   0          73s
pi@raspberrypi:~ $ kubectl describe pod -n kube-system storage-provisioner 
Name:         storage-provisioner
Namespace:    kube-system
Priority:     0
Node:         raspberrypi/192.168.0.18
Start Time:   Sat, 22 Aug 2020 04:16:20 +0100
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              integration-test=storage-provisioner
Annotations:  Status:  Pending
IP:           192.168.0.18
IPs:
  IP:  192.168.0.18
Containers:
  storage-provisioner:
    Container ID:  
    Image:         gcr.io/k8s-minikube/storage-provisioner-arm64:v2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /storage-provisioner
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from storage-provisioner-token-ld8jr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp
    HostPathType:  Directory
  storage-provisioner-token-ld8jr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  storage-provisioner-token-ld8jr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                From                  Message
  ----     ------            ----               ----                  -------
  Warning  FailedScheduling  85s (x2 over 85s)  default-scheduler     0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled         82s                default-scheduler     Successfully assigned kube-system/storage-provisioner to raspberrypi
  Normal   Pulling           38s (x3 over 78s)  kubelet, raspberrypi  Pulling image "gcr.io/k8s-minikube/storage-provisioner-arm64:v2"
  Warning  Failed            37s (x3 over 77s)  kubelet, raspberrypi  Failed to pull image "gcr.io/k8s-minikube/storage-provisioner-arm64:v2": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/k8s-minikube/storage-provisioner-arm64:v2 not found: manifest unknown: Failed to fetch "v2" from request "/v2/k8s-minikube/storage-provisioner-arm64/manifests/v2".
  Warning  Failed            37s (x3 over 77s)  kubelet, raspberrypi  Error: ErrImagePull
  Normal   BackOff           1s (x5 over 77s)   kubelet, raspberrypi  Back-off pulling image "gcr.io/k8s-minikube/storage-provisioner-arm64:v2"
  Warning  Failed            1s (x5 over 77s)   kubelet, raspberrypi  Error: ImagePullBackOff
pi@raspberrypi:~ $ 

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

[afbjorklund]

Thanks for reporting this, looks like we "forgot" to build for other architectures after bumping from v1.8.1 to v2 :

0538ddd

Error response from daemon: manifest for gcr.io/k8s-minikube/storage-provisioner-arm64:v2 not found: manifest unknown: Failed to fetch "v2" from request "/v2/k8s-minikube/storage-provisioner-arm64/manifests/v2".

In the future, we should move to a multi-arch manifest: #6007 (comment)

We also don't have any verification of arm64 architecture in CI: #6280

If you want to build it locally, you can make storage-provisioner-image

But it should have been a part of releasing the new storage provisioner

[afbjorklund]

Possibly related to ghost issue #8803

[afbjorklund]

Also missing for the new v3 version

[afbjorklund]

We should make it multi-arch: #9061

[afbjorklund]

Fixed in PR #9334