Enabling bidirectional mounts on pod kills docker

[Madhu-1]

Enabling bidirectional mounts on pods kills docker.
deployed kubernetes using minikube with VirtualBox driver with (CPUs=4, Memory=5000MB, Disk=80000MB)

• pod list output

[🎩︎]mrajanna@localhost rook $]kubectl get po -nsmoke-ns-system
NAME                                READY   STATUS              RESTARTS   AGE
csi-cephfsplugin-cg764              1/2     RunContainerError   0          85s
csi-cephfsplugin-provisioner-0      1/2     ErrImagePull        0          85s
csi-rbdplugin-attacher-0            1/1     Running             0          85s
csi-rbdplugin-provisioner-0         1/3     ImageInspectError   0          85s
csi-rbdplugin-vw4ch                 1/2     RunContainerError   0          85s
rook-ceph-agent-kqpzk               1/1     Running             0          85s
rook-ceph-operator-7f5858cb-ngrd5   1/1     Running             0          87s
rook-discover-dqdjh                 1/1     Running             0          85s

• docker ps output

$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

• kubectl describe output

  kubectl describe po/csi-rbdplugin-vw4ch -nsmoke-ns-system
Name:               csi-rbdplugin-vw4ch
Namespace:          smoke-ns-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Wed, 10 Apr 2019 10:12:56 +0530
Labels:             app=csi-rbdplugin
                    controller-revision-hash=8469978b6d
                    pod-template-generation=1
Annotations:        <none>
Status:             Running
IP:                 10.0.2.15
Controlled By:      DaemonSet/csi-rbdplugin
Containers:
  driver-registrar:
    Container ID:  docker://b0c4a874b40af193951955f0673faf1b26b926d490f3500384d2d48922d2d6f3
    Image:         quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
    Image ID:      docker-pullable://quay.io/k8scsi/csi-node-driver-registrar@sha256:ffecfbe6ae9f446e5102cbf2c73041d63ccf44bcfd72e2f2a62174a3a185eb69
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=/csi/csi.sock
      --kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock
    State:          Running
      Started:      Wed, 10 Apr 2019 10:13:03 +0530
    Ready:          True
    Restart Count:  0
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
  csi-rbdplugin:
    Container ID:  docker://60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c
    Image:         quay.io/cephcsi/rbdplugin:v1.0.0
    Image ID:      docker-pullable://quay.io/cephcsi/rbdplugin@sha256:893f9d27d3abfb3a4a263566f638af6b98c0660ad684f94ecd4e0586a90c05ec
    Port:          <none>
    Host Port:     <none>
    Args:
      --nodeid=$(NODE_ID)
      --endpoint=$(CSI_ENDPOINT)
      --v=5
      --drivername=rbd.csi.ceph.com
      --containerized=true
      --metadatastorage=k8s_configmap
    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: open /var/run/docker/runtime-runc/moby/60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c/state.json: no such file or directory: unknown
      Exit Code:    128
      Started:      Wed, 10 Apr 2019 10:13:53 +0530
      Finished:     Wed, 10 Apr 2019 10:13:53 +0530
    Ready:          False
    Restart Count:  0
    Environment:
      HOST_ROOTFS:    /rootfs
      NODE_ID:         (v1:spec.nodeName)
      POD_NAMESPACE:  smoke-ns-system (v1:metadata.namespace)
      CSI_ENDPOINT:   unix:///csi/csi.sock
    Mounts:
      /csi from plugin-dir (rw)
      /dev from host-dev (rw)
      /lib/modules from lib-modules (ro)
      /rootfs from host-rootfs (rw)
      /sys from host-sys (rw)
      /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/ from plugin-mount-dir (rw)
      /var/lib/kubelet/pods from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/rbd.csi.ceph.com
    HostPathType:  DirectoryOrCreate
  plugin-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  Directory
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  Directory
  host-dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  
  host-rootfs:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
  host-sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  rook-csi-rbd-plugin-sa-token-k2rnr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-csi-rbd-plugin-sa-token-k2rnr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  2m1s  default-scheduler  Successfully assigned smoke-ns-system/csi-rbdplugin-vw4ch to minikube
  Normal   Pulling    2m    kubelet, minikube  pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Pulled     114s  kubelet, minikube  Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Created    114s  kubelet, minikube  Created container
  Normal   Started    114s  kubelet, minikube  Started container
  Normal   Pulling    114s  kubelet, minikube  pulling image "quay.io/cephcsi/rbdplugin:v1.0.0"
  Normal   Pulled     64s   kubelet, minikube  Successfully pulled image "quay.io/cephcsi/rbdplugin:v1.0.0"
  Normal   Created    64s   kubelet, minikube  Created container
  Warning  Failed     64s   kubelet, minikube  Error: failed to start container "csi-rbdplugin": Error response from daemon: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/csi-rbdplugin/state.json: no such file or directory: unknown

• kubelet service status

 kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2019-04-10 04:39:41 UTC; 7min ago
     Docs: http://kubernetes.io/docs/
 Main PID: 3381 (kubelet)
    Tasks: 22 (limit: 4915)
   CGroup: /system.slice/kubelet.service
           └─3381 /usr/bin/kubelet --allow-privileged=true --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests

Apr 10 04:46:53 minikube kubelet[3381]: E0410 04:46:53.533805    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530571    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530706    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531381    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531477    3381 pod_workers.go:190] Error syncing pod b7b01c12-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "1fc8c7d721cae6017c0db69dc384eb0dcb08e35a0d7cf345ace875a6d3e9da33" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531816    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531983    3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.171467    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.174451    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.178389    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory

• journalctl.logs

-- Logs begin at Wed 2019-04-10 04:38:40 UTC, end at Wed 2019-04-10 04:48:41 UTC. --
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531469    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531745    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:05 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.535691    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.536757    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.172876    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.175689    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.181834    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:10 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529817    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529861    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.532928    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533038    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533863    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533918    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877333    3381 remote_image.go:112] PullImage "ceph/ceph:v14.2.0-20190319" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877367    3381 kuberuntime_image.go:51] Pull image "ceph/ceph:v14.2.0-20190319" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877456    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877472    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "version" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.878328    3381 remote_image.go:112] PullImage "quay.io/k8scsi/csi-snapshotter:v1.0.1" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879367    3381 kuberuntime_image.go:51] Pull image "quay.io/k8scsi/csi-snapshotter:v1.0.1" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879508    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879876    3381 remote_image.go:87] ImageStatus "quay.io/cephcsi/rbdplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879907    3381 kuberuntime_image.go:87] ImageStatus for image {"quay.io/cephcsi/rbdplugin:v1.0.0"} failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879938    3381 kuberuntime_manager.go:749] container start failed: ImageInspectError: Failed to inspect image "quay.io/cephcsi/rbdplugin:v1.0.0": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879950    3381 remote_image.go:112] PullImage "quay.io/cephcsi/cephfsplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879966    3381 kuberuntime_image.go:51] Pull image "quay.io/cephcsi/cephfsplugin:v1.0.0" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880010    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880729    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: [failed to "StartContainer" for "csi-snapshotter" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: , failed to "StartContainer" for "csi-rbdplugin" with ImageInspectError: "Failed to inspect image \"quay.io/cephcsi/rbdplugin:v1.0.0\": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: ]
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.882122    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/shim.sock" debug=false pid=9076
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim reaped" id=8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987516753Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987558237Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011173432Z" level=error msg="8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a cleanup: failed to delete container from containerd: no such container"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011222505Z" level=error msg="Handler for POST /v1.38/containers/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012419    3381 remote_runtime.go:213] StartContainer "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a" from runtime service failed: rpc error: code = Unknown desc = failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012487    3381 kuberuntime_manager.go:749] container start failed: RunContainerError: failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012509    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with RunContainerError: "failed to start container \"8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a\": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106412    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106462    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107038    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107065    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107828    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107858    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.108944    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.109941    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530818    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530911    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:15 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.177455    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.183538    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.189990    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531697    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531827    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:20 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530788    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530863    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530978    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531023    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531825    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531882    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530237    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530391    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531630    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531691    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532512    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532565    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.179720    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.207174    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.229088    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531689    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531840    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.238:86): table=filter family=2 entries=41
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.238:86): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.238:86): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=nat family=2 entries=62
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.242:87): table=nat family=2 entries=62
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.242:87): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.242:87): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533658    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533779    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530741    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530917    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:30 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.531959    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.532120    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530760    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530916    3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532209    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532323    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:35 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.174051    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.177904    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.187510    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.530998    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531070    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531773    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531809    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532157    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532203    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531047    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531176    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535568    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535734    3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.554986    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555594    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555118    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555737    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.556946    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.557064    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:40 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.539856    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.540295    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537196    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537281    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:45 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530651    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530799    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.177820    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.189574    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.198624    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530807    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530952    3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530410    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530630    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532267    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532593    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533392    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533490    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536147    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536487    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534434    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534826    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536539    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536712    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.545519    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.546282    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557738    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557931    3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.535959    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.536385    3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.539699    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.540229    3381 pod_workers.go:190] Error syncing pod b7e6760f-5b4a-11e9-ac49-08002742bbfb ("kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "8dccb241058a7c51d45d93f4a6f58d268d2d8a7b8b6dd9f44a2a24b6000b75b4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530768    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530873    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.531639    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.532034    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:55 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.174345    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.187741    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.194960    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.531670    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.532533    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:57 minikube audit[9177]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube kernel: audit: type=1325 audit(1554871497.289:88): table=filter family=2 entries=41
Apr 10 04:44:57 minikube kernel: audit: type=1300 audit(1554871497.289:88): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube kernel: audit: type=1327 audit(1554871497.289:88): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=nat family=2 entries=62

[tstromberg]

I'm not yet familiar with bi-directional mounts. Any chance you could post instructions to reproduce this bug?

[Madhu-1]

https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml#L71 this is the bi-directional mount used in the pod template
the full template is here https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml

[kfox1111]

I'm seeing something similar maybe. I fired up minikube 1.1.1 and tried to load in a csi driver of my own and odd things start happening immediately after the driver registers. containers fail to start complaining /etc/resolv.conf doesn't exist anymore. logging onto the host, and catting it, says it doesn't exist either. very strange. I confirmed it doesn't do this outside of minikube. Should I file this as a separate issue, or keep it as part of this one?

[giovanism]

@Madhu-1 Have you enabled the mount propagation?

• https://github.com/kubernetes-csi/docs/blob/735f1ef4adfcb157afce47c64d750b71012c8151/book/src/Setup.md#enabling-mount-propagation
• https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation

@tstromberg , Is there any way to pass MountFlags=shared to docker systemd unit file inside minikube?

[kfox1111]

Still seeing this with minikube 1.2.0.
Reproducer:
minikube start --feature-gates CSIInlineVolume=true
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-csidriverinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-daemonset.yaml

kubectl get pods #shows the pod never fully starting. You start to get weird errors like:
minikube ssh cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
ssh: Process exited with status 1

[kfox1111]

minikube ssh
sudo su -
mkdir -p /etc/systemd/system/docker.service.d/
(echo '[Service]'; echo 'MountFlags=shared') > /etc/systemd/system/docker.service.d/mount_flags.conf
 systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
systemctl show docker | grep Mount
#MountFlags=shared
exit
exit
#no workie
minikube stop
minikube start
#no workie
#/etc/systemd/system/docker.service.d/mount_flags.conf is missing after reboot.

[kfox1111]

Ah.... I found this: opencontainers/runc#1961
inspecting, it looks like it is the issue:

# mount | grep root
rootfs on / type rootfs (rw)
# df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay            0     0     0    - /
# 

[kfox1111]

I can confirm it with cri-o too:

minikube start -p crio --container-runtime=cri-o 
#confirm that it is cri-o based
minikube -p crio ssh ps ax | grep crio
 1988 ?        Ssl    0:13 /usr/bin/crio --log-level=debug --insecure-registry 10.96.0.0/12
 2860 ?        Ssl    0:00 /usr/libexec/crio/conmon --syslog -c 4e35107dd63ff5eef5b0adf9d6eb442f6a88429c40ce702523dae2a22ea99bdf -n k8s_POD_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2035f37c8a742e46
 2891 ?        Ssl    0:00 /usr/libexec/crio/conmon --syslog -c 085c98ce9552d32c5d8d0966a72685fd6753e547ddbea7f9d93acd3d04ac8b54 -n k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2
#check the mount
minikube -p crio ssh -- df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs             0     0     0    - /
#apply the driver
kubectl apply -f git/csi-driver-image-populator/deploy/image/csi-image-daemonset.yaml
#wait a bit for it to sort of start/crash
minikube -p crio ssh -- df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay            0     0     0    - /

[kfox1111]

Is there any way to switch the root filesystem away from rootfs?