Document how to add corporate root SSL certificates

[isuftin]

This is a question/feature request.

I am currently operating minikube on a network that requires all SSL traffic to be intercepted using a root SSL certificate. Minikube is using the Virtualbox driver.

Using docker-machine and boot2docker, I was able to add my root certificate to I need to be able to append my SSL root cert to the /var/lib/boot2docker/certs directory. This does not work with the minikube ISO. What does work is appending my certificate to /etc/ssl/certs/ca-certificates.crt

I create a bootlocal.sh for this. This is the command I run from my host:

read -d '' String <<"EOF"
#!/bin/bash
grep -q '<single line from root cert>' /etc/ssl/certs/ca-certificates.crt
EXIT_CODE=\\$?
if [ \\$EXIT_CODE -ne 0 ]; then
  curl http://<rootcert location>/root.cer | sudo tee -a /etc/ssl/certs/ca-certificates.crt
  sudo systemctl restart docker.service
fi
EOF
echo "echo "\""${String}"\"" | sudo tee -a /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh" | minikube ssh 

This works fine until I restart the VM. It looks like what's happening is that bootlocal may be running first or not at all because when the machine restarts, ca-certificates.crt is back to the original state without my certificate in it.

How should I add my root SSL cert into the VM so Docker can pull from registries?

minikube version: v0.18.0
OS: OSX 10.10.5
Drivername: Virtualbox
ISO: minikube-v0.18.0.iso
Install Tools: homebrew

What happened:

Corporate root SSL certificate does not exist on the VM and is not picked up by the Docker daemon

What you expected to happen:

I expect to be able to add my root SSL certificate to the VM and have Docker be able to pick it up to validate TLS communication

How to reproduce it (as minimally and precisely as possible):

$ minikube start

< run above commandset to pull down any SSL cert >

$ minikube stop && minikube start

< verify that SSL cert does not exist in ca-certificates.crt >

[r2d4]

I'm not exactly sure why this method isn't working, since it seems like it should. Either way, we should expose something similar to /var/lib/boot2docker/certs to auto load self-signed certs.

[isuftin]

@r2d4 Are you able to tell whether bootlocal.sh runs before or after the certificate configuration happens? If before, there's the issue.

I'd love to be able to just dump my cert into /var/lib/boot2docker/certs and call it a day like I do with docker-machine

[isuftin]

@r2d4 The other issue would be that /var/lib/boot2docker/certs gets wiped on VM reboot, so that would need to be persisted or maybe copied in from /data/certs or /Users/[...]. Having it persist would be preferable.

[calaniz]

The latest minikube VM isn't actually based off of boot2docker any more. You can find the buildroot documentation online.

What I did was add the appropriate crt to both the /etc/ssl/certs directory and the /etc/ssl/certs/ca-certificates.crt by adding them to the overlay in the current repo. You can find it at deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/. Create a ssl directory and drop your crts in.

[isuftin]

@calaniz I will give this a shot.

[justjwheelin]

For what it's worth, I tried following the advice of @calaniz above, but it didn't work for me on boot. I dropped my certs into /etc/ssl/certs/ and appended them to ca-certificates.crt, but the system isn't picking them up by default, e.g. running curl https://www.google.com. If I specify the path manually (--cacert /etc/ssl/certs/ca-certificates.crt), then it works as expected.

Trying to figure out a way to get this to work in Buildroot.

[justjwheelin]

Hmm, so I tried doing this again today and I think there might have been an issue with the certs I had on my local system. Reinstalling the ca-certificates package on my own system, rebuilding the ISO with the Makefiles, and running it in Minikube solved this issue. So it seems like @calaniz's solution also works for me after all.

Thanks for your help in the Slack yesterday, @r2d4!

[chino]

Is there going to be an easy way to do this at install/runtime without the need to rebuild ISOs and other steps?

[ztripez]

We are unable to run our docker repo without a cert and it's not exposed to the internet, iow we must use a self signed cert.
If there is no way to add the cert to minikube without building a new ISO makes it very hard for us to using minikube/kubernetes.

[dkirrane]

Could someone provide the steps to create the minikube ISO with my certs and how to pass it to minikube start?

[justjwheelin]

@dkirrane I actually forgot I was supposed to write documentation for this. A few months ago, I added a flag to Minikube that lets you specify a remote volume for generating the ISO (if you're making it from scratch).

See: #1657

Unfortunately, I no longer have access to the workstation / network where I was doing this development work. But if I remember right, you should be able to use the ISO_DOCKER_EXTRA_ARGS environment variable to specify a path to a volume, like…

$ ISO_DOCKER_EXTRA_ARGS='-v /path/to/certs/:/etc/ssl/certs/

Hopefully this is helpful for you. Sorry I can't test this further anymore.

[ChristofferNicklassonLenswayGroup]

Hi,

Docker wants to have the certs for repository's in /etc/docker/certs.d/site/ca.crt
Would like to see that path being persistent.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale