|
| 1 | +# (Experimental) NVIDIA GPU support in minikube |
| 2 | + |
| 3 | +minikube has experimental support for using NVIDIA GPUs on Linux. |
| 4 | + |
| 5 | +## Using NVIDIA GPUs on minikube on Linux with `--vm-driver=kvm2` |
| 6 | + |
| 7 | +When using NVIDIA GPUs with the kvm2 vm-driver. We passthrough spare GPUs on the |
| 8 | +host to the minikube VM. Doing so has a few prerequisites: |
| 9 | + |
| 10 | +- You must install the [kvm2 driver](drivers.md#kvm2-driver). If you already had |
| 11 | + this installed make sure that you fetch the latest |
| 12 | + `docker-machine-driver-kvm2` binary that has GPU support. |
| 13 | + |
| 14 | +- Your CPU must support IOMMU. Different vendors have different names for this |
| 15 | + technology. Intel calls it Intel VT-d. AMD calls it AMD-Vi. Your motherboard |
| 16 | + must also support IOMMU. |
| 17 | + |
| 18 | +- You must enable IOMMU in the kernel: add `intel_iommu=on` or `amd_iommu=on` |
| 19 | + (depending to your CPU vendor) to the kernel command line. Also add `iommu=pt` |
| 20 | + to the kernel command line. |
| 21 | + |
| 22 | +- You must have spare GPUs that are not used on the host and can be passthrough |
| 23 | + to the VM. These GPUs must not be controlled by the nvidia/nouveau driver. You |
| 24 | + can ensure this by either not loading the nvidia/nouveau driver on the host at |
| 25 | + all or assigning the spare GPU devices to stub kernel modules like `vfio-pci` |
| 26 | + or `pci-stub` at boot time. You can do that by adding the |
| 27 | + [vendorId:deviceId](https://pci-ids.ucw.cz/read/PC/10de) of your spare GPU to |
| 28 | + the kernel command line. For ex. for Quadro M4000 add `pci-stub.ids=10de:13f1` |
| 29 | + to the kernel command line. Note that you will have to do this for all GPUs |
| 30 | + you want to passthrough to the VM and all other devices that are in the IOMMU |
| 31 | + group of these GPUs. |
| 32 | + |
| 33 | +- Once you reboot the system after doing the above, you should be ready to you |
| 34 | + GPUs with kvm2. Run the following command to start minikube: |
| 35 | + ``` |
| 36 | + minikube start --vm-driver kvm2 --gpu |
| 37 | + ``` |
| 38 | + This command will check if all the above conditions are satisfied and |
| 39 | + passthrough spare GPUs found on the host to the VM. |
| 40 | + |
| 41 | + If this succeeded, run the following commands: |
| 42 | + ``` |
| 43 | + minikube addons enable nvidia-gpu-device-plugin |
| 44 | + minikube addons enable nvidia-driver-installer |
| 45 | + ``` |
| 46 | + This will install the NVIDIA driver on the VM. |
| 47 | + |
| 48 | +- If everything succeeded, you should be able to see `nvidia.com/gpu` in the |
| 49 | + capacity: |
| 50 | + ``` |
| 51 | + kubectl get nodes -ojson | jq .items[].status.capacity |
| 52 | + ``` |
| 53 | + |
| 54 | +### Where can I learn more about GPU passthrough? |
| 55 | +See the excellent documentation at |
| 56 | +https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF |
| 57 | + |
| 58 | +### Why are so many manual steps required to use GPUs with kvm2 on minikube? |
| 59 | +These steps require elevated privileges which minikube doesn't run with and they |
| 60 | +are disruptive to the host, so we decided to not do them automatically. |
| 61 | + |
| 62 | + |
| 63 | +## Using NVIDIA GPU on minikube on Linux with `--vm-driver=none` |
| 64 | + |
| 65 | +NOTE: This approach used to expose GPUs here is different than the approach used |
| 66 | +to expose GPUs with `--vm-driver=kvm2`. Please don't mix these instructions. |
| 67 | + |
| 68 | +- Install minikube. |
| 69 | + |
| 70 | +- Install the nvidia driver, nvidia-docker and configure docker with nvidia as |
| 71 | + the default runtime. See instructions at |
| 72 | + https://github.com/NVIDIA/nvidia-docker |
| 73 | + |
| 74 | +- Start minikube: |
| 75 | + ``` |
| 76 | + minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost |
| 77 | + ``` |
| 78 | + |
| 79 | +- Install NVIDIA's device plugin: |
| 80 | + ``` |
| 81 | + kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.10/nvidia-device-plugin.yml |
| 82 | + ``` |
| 83 | + |
| 84 | + |
| 85 | +## Why does minikube not support NVIDIA GPUs on macOS? |
| 86 | +VM drivers supported by minikube for macOS doesn't support GPU passthrough: |
| 87 | +- [mist64/xhyve#108](https://github.com/mist64/xhyve/issues/108) |
| 88 | +- [moby/hyperkit#159](https://github.com/moby/hyperkit/issues/159) |
| 89 | +- [VirtualBox docs](http://www.virtualbox.org/manual/ch09.html#pcipassthrough) |
| 90 | + |
| 91 | +Also: |
| 92 | +- For quite a while, all Mac hardware (both laptops and desktops) have come with |
| 93 | + Intel or AMD GPUs (and not with NVIDIA GPUs). Recently, Apple added [support |
| 94 | + for eGPUs](https://support.apple.com/en-us/HT208544), but even then all the |
| 95 | + supported GPUs listed are AMD’s. |
| 96 | + |
| 97 | +- nvidia-docker [doesn't support |
| 98 | + macOS](https://github.com/NVIDIA/nvidia-docker/issues/101) either. |
| 99 | + |
| 100 | + |
| 101 | +## Why does minikube not support NVIDIA GPUs on Windows? |
| 102 | +minikube suppports Windows host through Hyper-V or VirtualBox. |
| 103 | + |
| 104 | +- VirtualBox doesn't support PCI passthrough for [Windows |
| 105 | + host](http://www.virtualbox.org/manual/ch09.html#pcipassthrough). |
| 106 | + |
| 107 | +- Hyper-V supports DDA (discrete device assignment) but [only for Windows Server |
| 108 | + 2016](https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment) |
| 109 | + |
| 110 | +Since the only possibility of supporting GPUs on minikube on Windows is on a |
| 111 | +server OS where users don't usually run minikube, we haven't invested time in |
| 112 | +trying to support NVIDIA GPUs on minikube on Windows. |
| 113 | + |
| 114 | +Also, nvidia-docker [doesn't support |
| 115 | +Windows](https://github.com/NVIDIA/nvidia-docker/issues/197) either. |
0 commit comments