docker driver IP address not reachable from outside

[MoeHamdan]

I have noticed that you are advising to use docker driver instead of none, so I decided to use that as it seems promising. However when I use it I cannot bind it to the VM's IP neither could I make it take a static IP of what I need. It just creates some random IP address which will not work for me as all my work is outside the VM and I need to access docker and kubernetes cluster from outside the VM.

[afbjorklund]

Deploying Kubernetes for access outside of the local machine isn't really something that minikube is designed to do... It would actually have been better if the "none" driver would only bind to localhost and using a different driver for connecting to an external VM, such as the "generic" driver [#4733]. Otherwise we leave all setting up of the docker enviroment and kubernetes configuration to the user.

Related: #4313

[afbjorklund]

I'm assuming here that the "random IP" is the address of the docker container, like 172.17.0.3 ?

https://docs.docker.com/network/bridge/

[afbjorklund]

The "problem" with using the none driver is that it leaves the actual client unconfigured. It just configures the root account on the VM, which then isn't actually used for the actual development. Not that using the master node for development is great either, mostly because of the limited environment but also for teaching the anti-pattern of doing things locally as the root user. But OK for "training" (maybe)

So the user has to set up their DOCKER_HOST and their KUBECONFIG manually instead. Assuming here that they want to handle the SSH keys and setting up the VM, rather than having minikube do that. So basically all minikube is actually doing in this scenario is being a thin wrapper to kubeadm. Because the final setup, and using things like "docker-env" or "dashboard" doesn't really work either.

[afbjorklund]

It would have been great to add this aspect, to the discussion about minikube drivers in #8135

[MoeHamdan]

Hello @afbjorklund
Thanks for all this information. Yes I am getting this 172.17.0.3 or 172.17.0.2. Here is my thing, I have ubuntu custom VM and I use minikube on it, then on my windows machine I have scripts that pulls/build/deploys my microservices from git. What I currently do is that I use none driver to make this work. Is there a better way of doing this ?

[MoeHamdan]

I use my script to use docker build and kubectl to apply the yml configuration. So actually my script does not use minikube commands.

[afbjorklund]

Here is my thing, I have ubuntu custom VM and I use minikube on it, then on my windows machine I have scripts that pulls/build/deploys my microservices from git. What I currently do is that I use none driver to make this work. Is there a better way of doing this ?

Thank you for describing, "normally" what would happen is that minikube would manage your VM for you and run a more minimal custom OS. The promise is that minikube start would handle all three aspects, in your scenario you still have to do two of them manually:

1. Start VM, install Linux, install Docker
2. Install Kubernetes and start it (kubeadm)
3. Configure local clients, docker/kubectl

Which is fine if you need the customization, but otherwise a bit complicated for "just running kubernetes". I guess which parts of Ubuntu are missing in our Buildroot is another discussion, but one we are having as we are currently both of them to newer releases.

Meaning to write a little longer about this...

I use my script to use docker build and kubectl to apply the yml configuration. So actually my script does not use minikube commands.

In order to run those scripts, you will need to configure your environment. Otherwise they will try to contact things on your (windows) localhost. So there's usually some custom scripts to re-do the things from "docker-env" and "update-context" ?

The theory here is that it should be able for the user to just run minikube start on their Windows.

If you don't want minikube to provide a VM for you (using Hyper-V or whatever), then there should be a driver where you can just input the SSH parameters to it. That's the awkwardly named "generic". It could even be physical, or in a private cloud*.

* I'm assuming here that for the public cloud you would just start using an already available cluster.

[afbjorklund]

Also, you would be surprised how many people have problems installing docker and kubectl :-)

Which was why they were "integrated" with minikube, so you can just install one tool for all of it.

[MoeHamdan]

Well I use it for development not for production. The thing is it easier for our team to have a local dev environment with scripts deploying all or the needed microservices quickly. I used to use hyper-v and vmware driver but ended up with networking problems as I need a specific IP for the VM (also it should match my local network subnet mask) so I went with custom VM so I could have more control over it, and actually it is quite simple with none driver you just install docker then install minikube and all works. Also one final thing with the none I had an extra option which is the ability to delete minikube cluster without losing the images from docker. This way I can just reinstall minikube and apply the yml files without building all the code again. I have 50+ microservices.

[afbjorklund]

The images should be possible to cache (to not have to rebuild them or anything) between runs, but otherwise these kinds of legacy constraints (custom IPs) and such is why the custom option exists...

There are no fundamental differences between these drivers, by the way. It is kubeadm that is doing all the heavy lifting, so using the "none" driver is (or at least it should be) just a minimal wrapper for it.

And thanks for sharing your use case and scenario!

We should gather all of these in the documentation.