Conflict between docker config and docker service with ssh driver

[aishwarya-senthilkumar]

Unable to start, stop or delete minikube

minikube start --driver=ssh --ssh-ip-address=10.186.72.184 --native-ssh=false --service-cluster-ip-range='192.168.115.0/24' --extra-config=kubeadm.pod-network-cidr='192.168.114.0/24' --extra-config=kubeadm.pod-network-cidr=192.168.120.0/22 --cni=bridge --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
😄  minikube v1.29.0 on Darwin 12.3.1
    ▪ KUBECONFIG=/Users/sai/.kube/config
✨  Using the ssh driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Deleting "minikube" in ssh ...
🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

sai@saiEMD6M ~ % minikube stop
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...

❌  Exiting due to GUEST_STOP_TIMEOUT: Temporary Error: stop: containers: docker: docker ps -a --filter=name=k8s_ --format=: Process exited with status 1
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?


╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                       │
│    😿  If the above advice does not help, please let us know:                                                         │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                                                       │
│                                                                                                                       │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                                │
│    Please also attach the following file to the GitHub issue:                                                         │
│    - /var/folders/n1/7_bs1z291n9dx8vy62dg873h0000gq/T/minikube_stop_bc7f49f349e822641d09c5fb48f410196b822884_0.log    │
│                                                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

sai@saiEMD6M ~ % minikube delete
🔄  Uninstalling Kubernetes v1.25.3 using kubeadm ...
E0404 10:17:19.593080   13318 delete.go:516] unpause failed: list paused: docker: docker ps --filter status=paused --filter=name=k8s_ --format={{.ID}}: Process exited with status 1
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
💣  error deleting profile "minikube": failed to delete cluster: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.25.3:$PATH" kubeadm reset --cri-socket /var/run/cri-dockerd.sock --force": Process exited with status 127
stdout:

stderr:
env: ‘kubeadm’: No such file or directory

DETAILS :

OS : MacOS Monterey
Version 12.3.1 (21E258)

sai@saiEMD6M ~ % minikube version
minikube version: v1.28.0
commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f

sai@saiEMD6M ~ % docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.2 (95914)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I managed to login to my VM, docker service is down

root@minikube:~# systemctl status docker
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─10-machine.conf
     Active: failed (Result: exit-code) since Mon 2023-04-03 21:00:01 PDT; 52min ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 158216 ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver overlay2 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=ssh --insecure-r>
   Main PID: 158216 (code=exited, status=1/FAILURE)
        CPU: 89ms

Apr 03 21:00:01 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 03 21:00:01 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 03 21:00:01 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 03 21:00:01 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 03 21:00:01 minikube systemd[1]: Failed to start Docker Application Container Engine.
lines 1-16/16 (END)

Is there something wrong with my minikube setup? or is it something simple I’m missing?

[aishwarya-senthilkumar]

root@minikube:~# sudo journalctl -eu docker
~
~
~
~
~
~
~
~
~
~
~
~
~
~
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168313]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168324]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168335]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168374]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Apr 04 00:08:51 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168384]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Apr 04 00:08:51 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168393]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 04 00:08:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
root@minikube:~#

[aishwarya-senthilkumar]

Issue:

unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)

There is a directive in /etc/docker/daemon.json, that is a duplicate of the one passed in ExecStart in /etc/systemd/system/docker.service.d/10-machine.conf (storage-driver)

Solution:

Inside vm, Remove --storage-driver overlay2 from /etc/systemd/system/docker.service.d/10-machine.conf
then sudo systemctl daemon-reload
then sudo systemctl restart docker

Then try running the same minikube start --driver=ssh ... on your machine

[afbjorklund]

The "ssh" driver is for advanced users, it seems there might be some conflict between the minikube docker config and what was installed on the VM before. I would suggest using another driver, unless your workaround is enough for you.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale