Added hyperkit options for enterprise VPN support #9

seborama · 2018-05-28T17:01:17Z

The purpose of these changes is to enhance Hyperkit support from the
minikube command line for better integration with enterprise networks
behind a VPN.

uuid: Provide VM UUID to restore MAC address (only supported with
Hyperkit driver).
vpnkitSock: Location of the VPNKit socket used for networking. If empty,
disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac
VPNKit connection, otherwise uses the specified VSock."
vsockPorts: List of guest VSock ports that should be exposed as sockets
on the host (Only supported on with hyperkit now).

The purpose of these changes is to enhance Hyperkit support from the minikube command line for better integration with enterprise networks behind a VPN. uuid: Provide VM UUID to restore MAC address (only supported with Hyperkit driver). vpnkitSock: Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac VPNKit connection, otherwise uses the specified VSock." vsockPorts: List of guest VSock ports that should be exposed as sockets on the host (Only supported on with hyperkit now). Note: tests pass but file: `vendor/github.com/google/certificate-transparency/go/x509/root_darwin.go` has to be edited to correct an issue - not committed since this is in the vendor directory.

praveenkumar

Overall looks good to me, we do need to test this piece and merge it.

praveenkumar · 2018-05-30T13:25:51Z

pkg/hyperkit/driver.go

@@ -103,7 +107,7 @@ func (d *Driver) Create() error {

 	isoPath := d.ResolveStorePath(isoFilename)
 	if err := d.extractKernel(isoPath); err != nil {
-		return err
+		return errors.Wrap(err, "extracting kernel")


praveenkumar · 2018-05-30T13:26:01Z

pkg/hyperkit/driver.go

@@ -175,25 +179,32 @@ func (d *Driver) Restart() error {

 // Start a host
 func (d *Driver) Start() error {
-	h, err := hyperkit.New("", "", filepath.Join(d.StorePath, "machines", d.MachineName))
+	h, err := hyperkit.New("", d.VpnKitSock, filepath.Join(d.StorePath, "machines", d.MachineName))


seborama · 2018-06-01T07:15:18Z

Hi,

With regards to testing, below are examples of ~/.minikube/machines/minikube/hyperkit.json with different minikube start options.

EXAMPLE 1

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 --uuid=8de16914-60d3-11e8-b5f4-784f438fc458 --hyperkit-vpnkit-sock=auto --hyperkit-vsock-ports="2376,8443,30000,8080" && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json:
Notice the values of the uuid, vsock, vsock_ports, vpnkit_sock. Also, the VM was given the same IP address as previously given for this uuid.
{"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"/Users/someone/Library/Containers/com.docker.docker/Data/s50","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"8de16914-60d3-11e8-b5f4-784f438fc458","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":true,"vsock_ports":[2376,8443,30000,8080],"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":6305,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50","-s","2:0,virtio-net","-U","8de16914-60d3-11e8-b5f4-784f438fc458","-s","3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080","-s","5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","6,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50 -s 2:0,virtio-net -U 8de16914-60d3-11e8-b5f4-784f438fc458 -s 3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080 -s 5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 6,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

EXAMPLE 2

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json:
A new IP was provided, no more vsock activation
{"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"4b8107f3-656a-11e8-9069-784f438fc457","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":false,"vsock_ports":null,"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":8607,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-net","-U","4b8107f3-656a-11e8-9069-784f438fc457","-s","2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","4,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 4b8107f3-656a-11e8-9069-784f438fc457 -s 2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

seborama · 2018-06-02T10:36:31Z

Sorry, I have found one problem with the parsing of ports. They're defined as int rather than uint16 by Hyperkit and the conversion sometimes fails.
Investigating options...

seborama · 2018-06-02T18:11:57Z

OK, it seems the problem may have been caused by me. I have 3 versions of the driver and 2 of minikube owing to the forked repos and one from homebrew.
I've re-sync'ed both minikube and the hyperkit driver from a fresh compilation and the problem has gone away. I can only assume that different versions caused a clash.

LalatenduMohanty · 2018-06-04T03:11:48Z

@seborama Thanks for the PR, it will be helpful for other projects (including Minishift) using hyperkit.

seborama · 2018-06-04T22:07:04Z

Hi @LalatenduMohanty / @dlorenc / @praveenkumar ,
More work would be necessary in a future PR to improve the experience. I'm trying to understand how the minikube VM IP is generated. I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct?
Could you confirm and enrich my understanding?
Several scenarios exist currently which lead to failure when starting minikube as the IP is not generated for the minikube - Hyperkit configuration.

praveenkumar · 2018-06-07T13:42:24Z

I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct?

Yes that's correct https://github.com/machine-drivers/docker-machine-driver-hyperkit/blob/master/pkg/hyperkit/network.go#L48-L65

praveenkumar · 2018-06-07T13:43:37Z

I can also see this PR is merged from minikube side so @seborama do you think we can merge it here also and then gradually built it by resolving the issues.

seborama · 2018-06-07T14:43:31Z

Hi Praveen,

Thanks for your reply.

I'd say so yes. The code itself works fine as far as I can tell.

The intricacies (or my understanding thereof 😛) come from how the Minikube VM's IP address is allocated by Hyperkit. Starting a new machine with vsock for the first time won't allocate an IP. The machine needs creating once then deleted and finally re-created with vsock and the uuid of the defunct machine. Awkward. I need to look into the Hyperkit code to gain a better understanding.

nanzhong · 2018-07-24T11:57:26Z

Is this PR ready to be merged over on this side? I currently make heavy use of this and and building the driver from source, but it would be nice to see these changes merged.

seborama · 2018-07-24T12:18:33Z

It is for this iteration.
I think that further enhancements can be part of separate PR's. For instance, we found that routing in the Minikube K8s cluster to external addresses is not as functional as with say VirtualBox for instance.

seborama · 2018-07-24T14:31:04Z

Hi @praveenkumar,

Is there anything outstanding on this PR?

Cheers

seborama · 2018-07-24T19:12:29Z

Hey @nanzhong ,

Currently the code from this PR exists both in the minikube repo (already merged by @dlorenc ) and this repo (this PR, pending).

This means that you can compile the drivers from the Minikube repo (use make driver and optionally make install install-hyperkit-driver or copy the file file manually from the out directory and apply the correct permissions).

HOWEVER... As documented at https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver, I've had a look at the binary distribution at https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit.

The binary has been updated from the minikube repo. So you don't need to compile it yourself if you don't want to, just follow the steps in the doc from the minikube project (above link).

Best of luck.

PS: details of the driver update as reported at https://storage.googleapis.com/minikube/:
(in short: yesterday)

<Contents>
<Key>releases/latest/docker-machine-driver-hyperkit</Key>
<Generation>1532362270225121</Generation>
<MetaGeneration>1</MetaGeneration>
<LastModified>2018-07-23T16:11:10.224Z</LastModified>
<ETag>"6c90be83e9882c5bd838e2bb8461dc59"</ETag>
<Size>26811748</Size>
</Contents>

nanzhong · 2018-07-24T19:16:14Z

A perfect, thanks!

seborama · 2018-08-23T18:57:04Z

Hi,
This PR is stale. I'm closing it.
Best regards.

seborama mentioned this pull request May 28, 2018

Added hyperkit options for enterprise VPN support kubernetes/minikube#2850

Merged

seborama force-pushed the master branch from f88bcd8 to 821747a Compare May 29, 2018 19:27

praveenkumar approved these changes May 30, 2018

View reviewed changes

seborama closed this Aug 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added hyperkit options for enterprise VPN support #9

Added hyperkit options for enterprise VPN support #9

seborama commented May 28, 2018 •

edited

Loading

praveenkumar left a comment

praveenkumar May 30, 2018

praveenkumar May 30, 2018

seborama commented Jun 1, 2018

seborama commented Jun 2, 2018

seborama commented Jun 2, 2018

LalatenduMohanty commented Jun 4, 2018

seborama commented Jun 4, 2018 •

edited

Loading

praveenkumar commented Jun 7, 2018

praveenkumar commented Jun 7, 2018

seborama commented Jun 7, 2018

nanzhong commented Jul 24, 2018

seborama commented Jul 24, 2018

seborama commented Jul 24, 2018

seborama commented Jul 24, 2018 •

edited

Loading

nanzhong commented Jul 24, 2018

seborama commented Aug 23, 2018

Added hyperkit options for enterprise VPN support #9

Added hyperkit options for enterprise VPN support #9

Conversation

seborama commented May 28, 2018 • edited Loading

praveenkumar left a comment

Choose a reason for hiding this comment

praveenkumar May 30, 2018

Choose a reason for hiding this comment

praveenkumar May 30, 2018

Choose a reason for hiding this comment

seborama commented Jun 1, 2018

seborama commented Jun 2, 2018

seborama commented Jun 2, 2018

LalatenduMohanty commented Jun 4, 2018

seborama commented Jun 4, 2018 • edited Loading

praveenkumar commented Jun 7, 2018

praveenkumar commented Jun 7, 2018

seborama commented Jun 7, 2018

nanzhong commented Jul 24, 2018

seborama commented Jul 24, 2018

seborama commented Jul 24, 2018

seborama commented Jul 24, 2018 • edited Loading

nanzhong commented Jul 24, 2018

seborama commented Aug 23, 2018

seborama commented May 28, 2018 •

edited

Loading

seborama commented Jun 4, 2018 •

edited

Loading

seborama commented Jul 24, 2018 •

edited

Loading