Cannoot access services via NodePort on MacOS with Docker driver

[marlon-sousa]

Hello,

I am facing issues accessing services via node port on Mac OS.

System information:

ProductName:    macOS
ProductVersion: 11.1
BuildVersion:   20C69

Ninikube version:

minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5

kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d3
25d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

docker version:

Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Minikube installed with homebrew version:

Homebrew 3.1.3
Homebrew/homebrew-core (git revision 9bc3198bef; last commit 2021-04-25)

minikub init command

minikube start --driver=docker --mount --mount-string $(PWD)/dev/volumes:/volumes

Steps to reproduce the issue:

1. On Mac OS, start minikube as above.
2. Make a simple deployment / service using mode port. Below I will post a yaml but this has also veen berified with other services I tried.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: wiremock-pv-volume
  namespace: queen
  labels:
    type: local
    app: wiremock
spec:
  storageClassName: manual
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/volumes/wiremock"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wiremock-pv-claim
  labels:
    app: wiremock
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi

-------
apiVersion: v1
kind: Service
metadata:
  name: wiremock
  labels:
    app: wiremock
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30080
  selector:
    app: wiremock

-------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wiremock
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wiremock
  template:
    metadata:
      labels:
        app: wiremock
    spec:
      containers:
        - name: wiremock
          image: rodolpheche/wiremock
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /home/wiremock
              name: wiremock-vol
      volumes:
        - name: wiremock-vol
          persistentVolumeClaim:
            claimName: wiremock-pv-claim

4. Up until here we are pretty good. The volume has been created, shared, everything working. kubectl get svc shows the following:

NAME       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
wiremock   NodePort   10.102.252.201   <none>        8080:30080/TCP   18h

issue

But I cannot access this service using port 30080, and here comes the issue:
If I try

minikube ip

I get 192.168.49.2 as response. This is what happens also on Linux. However, this ip is kind of "useless", in the sense that I can not ping it from terminal nor can I telnet or CURL it at port 30080 (or at any other port I have tried).
This is different from Linux, where I can either ping or telnet / CURL / PSQL / whatever I need at the appropriate ports and services work. on Mac OS, every telnet / CURL / psql I try on this IP hangs indefinitely.

I suppose this has to do somehow with the docker driver, but I couldn't find documentation on that nor could I find anything useful in logs.
Strangely, minikube ssh works flowlessly, as also works minikube service wiremock. More strange is that minikube tunnel also hangs indefinitely.

I cannot use minikube service tunelling for several reazons, and I am thinking what can be done to fix that. NodePort is an important feature of Minikube, so if the docker driver doesn't support that I would like to know.

As I have no errors outputs, I will post no logs here now, but I am whiling to do everything I can to help to track this, so feel free to ask me for extra logs.

Full output of failed command:

There is not failure of output commands, at least that I have seen.

Full output of minikube start command used, if not already included:

* minikube v1.19.0 on Darwin 11.1
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Updating the running docker "minikube" container ...
* Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
* Verifying Kubernetes components...
  - Using image kubernetesui/dashboard:v2.1.0
  - Using image kubernetesui/metrics-scraper:v1.0.4
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: default-storageclass, storage-provisioner, dashboard
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
* ```

[afbjorklund]

This is a known issue, Docker Desktop networking doesn't support ports. You will have to use minikube tunnel.

https://docs.docker.com/docker-for-mac/networking/#known-limitations-use-cases-and-workarounds

https://minikube.sigs.k8s.io/docs/handbook/accessing/

You could also run minikube in a VM, to get a proper IP. Such a driver can be selected with minikube start --vm

[afbjorklund]

Strangely, minikube ssh works flowlessly

minikube publishes the ssh port automatically

I cannot use minikube service tunelling for several reazons, and I am thinking what can be done to fix that. NodePort is an important feature of Minikube, so if the docker driver doesn't support that I would like to know.

There are other solutions to set up tunneling and networking, but nothing that is provided out-of-the-box.

[zhan9san]

Hi @marlon-sousa

This issue is similar to #9404
A custom service is required to be published on localhost/MacOS.

1. Expose specific port which app listens on. [Host Port]:[NodePort in Kubernetes Cluster]

minikube start --driver=docker --ports=30080:30080

2. Create Service

kubectl apply -f test.yml

test.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: metrics
      department: engineering
  replicas: 1
  template:
    metadata:
      labels:
        app: metrics
        department: engineering
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:2.0"
        env:
        - name: "PORT"
          value: "8080"

---
apiVersion: v1
kind: Service
metadata:
  name: my-np-service
spec:
  type: NodePort
  selector:
    app: metrics
    department: engineering
  ports:
  - protocol: TCP
    port: 8080
    nodePort: 30080

3. Verify Service

~/Source/k8s-test/svc ❯ docker port minikube                                                                                                         12:22:42
22/tcp -> 127.0.0.1:61438
2376/tcp -> 127.0.0.1:61439
30080/tcp -> 0.0.0.0:30080
32443/tcp -> 127.0.0.1:61440
5000/tcp -> 127.0.0.1:61441
8443/tcp -> 127.0.0.1:61442
~/Source/k8s-test/svc ❯ kubectl get svc                                                                                                              12:22:46
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP          5m3s
my-np-service   NodePort    10.104.99.25   <none>        8080:30080/TCP   4m48s
~/Source/k8s-test/svc ❯ curl http://127.0.0.1:30080                                                                                                  12:22:52
Hello, world!
Version: 2.0.0
Hostname: my-deployment-6ccc959b54-p4wst
~/Source/k8s-test/svc ❯ minikube ssh                                                                                                                 12:22:59
Last login: Mon Apr 26 04:19:34 2021 from 192.168.49.1
docker@minikube:~$ curl http://127.0.0.1:30080
Hello, world!
Version: 2.0.0
Hostname: my-deployment-6ccc959b54-p4wst

@afbjorklund
This issue brings me back to considering whether #11070 is necessary.
It seems --ports 8443:8443 can expose apiserver as well.

[afbjorklund]

@zhan9san But #11070 (and #8008) is about exporting the API server outside of localhost ?
Normally ssh and apiserver (port 8443) are only available to 127.0.0.1, for security reasons.

CONTAINER ID   IMAGE                                                         COMMAND                  CREATED         STATUS         PORTS                                                                                                                                  NAMES
5738d920d888   gcr.io/k8s-minikube/kicbase-builds:v0.0.20-1618262794-11055   "/usr/local/bin/entr…"   5 minutes ago   Up 5 minutes   127.0.0.1:49167->22/tcp, 127.0.0.1:49166->2376/tcp, 127.0.0.1:49165->5000/tcp, 127.0.0.1:49164->8443/tcp, 127.0.0.1:49163->32443/tcp   minikube

Using docker publishing for apps is not recommended, because it requires restarting the container.
So the ssh tunnels are (marginally) better, but a better ingress solution for crippled drivers is required...

---

I'm personally looking into NoRouter...
https://norouter.io/docs/examples/docker/

It would offer a HTTP proxy, for the cluster.
There are some other similar solutions.

[zhan9san]

@afbjorklund

1. It supports localhost as well. And it provides a workaround to this issue.

~ 14s ❯ minikube start --driver=docker --ports=127.0.0.1:30080:30080                                                                                 14:26:41
😄  minikube v1.19.0 on Darwin 10.15.7
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3132MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
~ 1m 32s ❯ docker port minikube                                                                                                                      14:28:15
22/tcp -> 127.0.0.1:51362
2376/tcp -> 127.0.0.1:51363
30080/tcp -> 127.0.0.1:30080
32443/tcp -> 127.0.0.1:51364
5000/tcp -> 127.0.0.1:51365
8443/tcp -> 127.0.0.1:51368

2. Using docker publishing for apps is not recommended. For this scenario, could you please explain why does it require restarting the container? Does it mean minikube delete/minikube start --driver=docker --ports=127.0.0.1:30080:30080
   Both docker and ingress can make application accessible outside cluster. If the user choose 'LoadBalancer', I noticed you mentioned minikube tunnel, it works well and there is no need to publish port when Minikube starts.

Minikube provides us a very convenient way to verify the application in localhost/development, minikube start/minikube delete. The cluster is ephemeral.

[zhan9san]

I agree with you, There are some other similar solutions.

Regarding NoRouter, it's a great tool providing mutual interconnectivity across multiple remote hosts.

Besides, using a one-time pod to access k8s cluster running in container is another choice.

~ ❯ minikube start --driver=docker
~ ❯ kubectl apply -f test.yml
~ ❯ minikube ip                                                                                                                                      15:24:40
192.168.49.2
~ ❯ kubectl run -it --rm --image=curlimages/curl --restart=Never one-time-curl -- http://192.168.49.2:30080                                          15:27:48
Hello, world!
Version: 2.0.0
Hostname: my-deployment-6ccc959b54-cwrb6
pod "one-time-curl" deleted

[afbjorklund]

Using docker publishing for apps is not recommended.
For this scenario, could you please explain why does it require restarting the container?

That's just how the docker publish works, it only takes affect when the container is started.

So when you add a new app, the new NodePort would have to be added to the publish list.

[zhan9san]

@afbjorklund
Thanks for you clarification. Yes, you're right.

My opinion is that a k8s cluster created by minikube is ephemeral, that's to say, once quick verification is done, destroy cluster as well. The scenario you mentioned will not exist.

Here is a bold idea, how about adding -P, --publish-all in minikube?
I'm not sure whether docker dynamically publishes ports or just when it starts?

[afbjorklund]

Here is a bold idea, how about adding -P, --publish-all in minikube?
I'm not sure whether docker dynamically publishes ports or just when it starts?

That just means to publish all ports that are exposed in the container image
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#expose

[zhan9san]

@afbjorklund

You are right.

Let's focus on the root cause, different from the Linux system, there is no bridge0 on Macos, and it makes container IP unreachable from host.

Fortunately, we can define the port range in both apiserver and docker publishing at the same time.
Restarting container would not be an issue.

Please see sample below. The small range is for performance.

~ ❯ minikube start --driver=docker --extra-config=apiserver.service-node-port-range=32760-32767 --ports=127.0.0.1:32760-32767:32760-32767            18:01:59
😄  minikube v1.19.0 on Darwin 10.15.7
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3132MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
    ▪ apiserver.service-node-port-range=32760-32767
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
~ 1m 23s ❯ kubectl get svc                                                                                                                           18:03:30
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP          66s
my-np-service   NodePort    10.101.214.134   <none>        8080:32760/TCP   13s
~ ❯ curl 127.0.0.1:32760                                                                                                                             18:04:31
Hello, world!
Version: 2.0.0
Hostname: my-deployment-6ccc959b54-z97pf

~ ❯ docker port minikube                                                                                                                             18:03:26
32763/tcp -> 127.0.0.1:32763
32766/tcp -> 127.0.0.1:32766
5000/tcp -> 127.0.0.1:58090
32761/tcp -> 127.0.0.1:32761
32762/tcp -> 127.0.0.1:32762
32443/tcp -> 127.0.0.1:58088
32760/tcp -> 127.0.0.1:32760
32764/tcp -> 127.0.0.1:32764
32765/tcp -> 127.0.0.1:32765
32767/tcp -> 127.0.0.1:32767
8443/tcp -> 127.0.0.1:58091
22/tcp -> 127.0.0.1:58085
2376/tcp -> 127.0.0.1:58089

[afbjorklund]

I don't want to open more ports, and I want to close down 2376 and 5000 whenever possible...

• 2376 is the legacy tcp port for DOCKER_HOST
• 5000 is the workaround for missing registry cert

This is not only an issue for the Docker Desktop drivers (Mac and Win), but also the SSH driver.

For now, it's a known shortcoming and it is being addressed by minikube tunnel (and ssh -L)

---

It could definitely be documented better:

I suppose this has to do somehow with the docker driver, but I couldn't find documentation on that nor could I find anything useful in logs.

Also interesting why tunneling doesn't work:

I cannot use minikube service tunelling for several reazons, and I am thinking what can be done to fix that. NodePort is an important feature of Minikube, so if the docker driver doesn't support that I would like to know.