Publish arm64 image for gcr.io/k8s-minikube/storage-provisioner

[wangxiyuan]

The exact command to reproduce the issue:
minikube start --vm-driver=none --kubernetes-version=v1.16.2 on an ARM machine

The full output of the command that failed:

The storage-provisioner pod is falied to start.

The output of the minikube logs command:

Back-off pulling image "gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1"

The operating system version:
Ubuntu 18.04 for aarch64

[sharifelgamal]

@wangxiyuan can you try this with minikube 1.6.1 and see if it's fixed?

[wangxiyuan]

No, The detail:

root@ubuntu:~# ./minikube-linux-arm64 status
host: 
kubelet: 
apiserver: 
kubeconfig: 

root@ubuntu:~# ./minikube-linux-arm64 start --vm-driver=none --kubernetes-version=v1.16.2
* minikube v1.6.1 on Ubuntu 18.04 (/arm64)
* Selecting 'none' driver from user configuration (alternates: [])
* Running on localhost (CPUs=4, Memory=16038MB, Disk=95733MB) ...
* OS release is Ubuntu 18.04.3 LTS
* Preparing Kubernetes v1.16.2 on Docker '18.09.7' ...
E1217 01:34:18.397568    5330 cache_images.go:85] CacheImage k8s.gcr.io/coredns-arm64:1.6.2 -> /root/.minikube/cache/images/k8s.gcr.io/coredns-arm64_1.6.2 failed: MANIFEST_UNKNOWN: "Failed to fetch \"1.6.2\" from request \"/v2/coredns-arm64/manifests/1.6.2\"."
E1217 01:34:18.772220    5330 cache_images.go:85] CacheImage gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1 -> /root/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner-arm64_v1.8.1 failed: MANIFEST_UNKNOWN: "Failed to fetch \"v1.8.1\" from request \"/v2/k8s-minikube/storage-provisioner-arm64/manifests/v1.8.1\"."
E1217 01:34:18.772290    5330 start.go:841] Error caching images:  Caching images for kubeadm: caching images: caching image /root/.minikube/cache/images/k8s.gcr.io/coredns-arm64_1.6.2: MANIFEST_UNKNOWN: "Failed to fetch \"1.6.2\" from request \"/v2/coredns-arm64/manifests/1.6.2\"."
  - kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
* Unable to load cached images: loading cached images: loading image k8s.gcr.io/coredns-arm64:1.6.2: stat /root/.minikube/cache/images/k8s.gcr.io/coredns-arm64_1.6.2: no such file or directory
* Pulling images ...
* Launching Kubernetes ... 
* Configuring local host environment ...
* 
! The 'none' driver provides limited isolation and may reduce system security and reliability.
! For more information, see:
  - https://minikube.sigs.k8s.io/docs/reference/drivers/none/
* 
! kubectl and minikube configuration will be stored in /root
! To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:
* 
  - sudo mv /root/.kube /root/.minikube $HOME
  - sudo chown -R $USER $HOME/.kube $HOME/.minikube
* 
* This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
* Waiting for cluster to come online ...
* Done! kubectl is now configured to use "minikube"

root@ubuntu:~# kubectl get pods --namespace kube-system
NAME                               READY   STATUS             RESTARTS   AGE
coredns-5644d7b6d9-fjsfq           1/1     Running            0          2m56s
coredns-5644d7b6d9-px82t           1/1     Running            0          2m56s
etcd-minikube                      1/1     Running            0          2m2s
kube-addon-manager-minikube        1/1     Running            0          108s
kube-apiserver-minikube            1/1     Running            0          111s
kube-controller-manager-minikube   1/1     Running            0          103s
kube-proxy-mrg29                   1/1     Running            0          2m57s
kube-scheduler-minikube            1/1     Running            0          110s
storage-provisioner                0/1     ImagePullBackOff   0          2m56s

root@ubuntu:~# kubectl describe pods storage-provisioner --namespace kube-system
Name:         storage-provisioner
Namespace:    kube-system
Priority:     0
Node:         minikube/192.168.1.203
Start Time:   Tue, 17 Dec 2019 01:35:01 +0000
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              integration-test=storage-provisioner
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","integration-test":"s...
Status:       Pending
IP:           192.168.1.203
IPs:
  IP:  192.168.1.203
Containers:
  storage-provisioner:
    Container ID:  
    Image:         gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /storage-provisioner
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from storage-provisioner-token-24vfq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp
    HostPathType:  Directory
  storage-provisioner-token-24vfq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  storage-provisioner-token-24vfq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  <unknown>             default-scheduler  Successfully assigned kube-system/storage-provisioner to minikube
  Normal   Pulling    106s (x4 over 3m14s)  kubelet, minikube  Pulling image "gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1"
  Warning  Failed     105s (x4 over 3m13s)  kubelet, minikube  Failed to pull image "gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1 not found
  Warning  Failed     105s (x4 over 3m13s)  kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    90s (x6 over 3m13s)   kubelet, minikube  Back-off pulling image "gcr.io/k8s-minikube/storage-provisioner-arm64:v1.8.1"
  Warning  Failed     78s (x7 over 3m13s)   kubelet, minikube  Error: ImagePullBackOff

[tstromberg]

I believe the root cause here is that we've never published an arm64 version of the storage-provisioner.

I don't think the storage-provisioner is particularly useful for the none driver, but we should probably publish one anyways.

[wangxiyuan]

@tstromberg yeah, it doesn't break my usage, while it's good to support.

BTW, And a question that why it requires storage-provisioner v1.8.1 but I specified k8s to v1.16.2

[tstromberg]

The storage provisioner is version independent - and actually hasn't been updated in years.

[afbjorklund]

Can we replace the storage-provisioner image with a multi-arch manifest list, perhaps ?

https://kubernetes.io/docs/concepts/containers/images/#building-multi-architecture-images-with-manifests

[afbjorklund]

I'm rewriting the Makefile slightly, to be able to cross-build the image for other architectures too.

Basically make storage-provisioner-image, which currently should build one for the local platform.

[afbjorklund]

Sample code for building multi-arch manifest, needs all the sub-arch images:

https://github.com/kubernetes/kubernetes/blob/master/test/images/image-util.sh

# This option is for running docker manifest command
export DOCKER_CLI_EXPERIMENTAL := enabled

[medyagh]

@wangxiyuan I just pushed arch images
for arm and arm64 for 1.8.1

The push refers to repository [gcr.io/k8s-minikube/storage-provisioner-arm]
0fc34639ef3d: Pushed 
v1.8.1: digest: sha256:09929559fe27a8127fcedee2b377e498ce305c1f661d75b39e31f7d61ad912f6 size: 528

The push refers to repository [gcr.io/k8s-minikube/storage-provisioner-arm64]
183043f23222: Pushed 
v1.8.1: digest: sha256:5093f94b04c41cbd7311bc55348e1cc5e86fc2f5dee7995ae6cd4795a0262d6a size: 528

[afbjorklund]

@medyagh : Thanks, if you could also push s390x and ppc64le we are good for now

Eventually (next storage-provisioner upgrade), we want to move to multi-arch and one manifest

[medyagh]

pushed s390x:

The push refers to repository [gcr.io/k8s-minikube/storage-provisioner-s390x]
6bd1d911bfe6: Pushed 
v1.8.1: digest: sha256:62fd641a2f44fdbad273b445b20e9d4d5ea5e2eccc623906886372e8d918a6a8 size: 528

The push refers to repository [gcr.io/k8s-minikube/storage-provisioner-ppc64le]
a3449ee948e9: Pushed 
v1.8.1: digest: sha256:211b94cd9157682b71bb97bc1bdebcd6b3706d8a66ffca4e2f8a07246a2e2e5a size: 528

[wangxiyuan]

thank you very much

[prankkelkar]

Thank you very much.