Skip to content
This repository was archived by the owner on May 12, 2021. It is now read-only.

Commit a5eb3aa

Browse files
author
Graham Whaley
committed
install: add Minikube installation instructions
Add instructions of how to setup and install Kata under Minikue. Fixes: #444 Signed-off-by: Graham Whaley <[email protected]>
1 parent df57d05 commit a5eb3aa

File tree

1 file changed

+240
-0
lines changed

1 file changed

+240
-0
lines changed

‎install/minikube-installation-guide.md

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# Installing Kata Containers in [Minikube](https://kubernetes.io/docs/setup/minikube/)
2+
3+
* [Installing Kata Containers in Minikube](#installing-kata-containers-in-minikube)
4+
* [Introduction](#introduction)
5+
* [Prerequisites](#prerequisites)
6+
* [Setting up Minikube](#setting-up-minikube)
7+
* [Checking for nested virtualization](#checking-for-nested-virtualization)
8+
* [Check Minikube is running](#check-minikube-is-running)
9+
* [Installing Kata Containers](#installing-kata-containers)
10+
* [Enabling Kata Containers](#enabling-kata-containers)
11+
* [Register the runtime](#register-the-runtime)
12+
* [Testing Kata Containers](#testing-kata-containers)
13+
* [Wrapping up](#wrapping-up)
14+
15+
## Introduction
16+
17+
[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s)
18+
cluster locally. It utilises running a single node Kubernetes stack in a local VM.
19+
20+
[Kata Containers](https://github.com/kata-containers) can be installed into a Minikube cluster using
21+
[kata-deploy](https://github.com/kata-containers/packaging/tree/master/kata-deploy).
22+
23+
This document details the pre-requisites, installation steps, and how to check
24+
the installation has been successful.
25+
26+
## Prerequisites
27+
28+
This installation guide has only been verified under a Minikube Linux installation, using the
29+
[kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver.
30+
31+
This installation guide may not work for MacOS installations of Minikube, due to the lack of
32+
nested virtualization support on that platform.
33+
34+
This installation guide has not been tested on a Windows installation.
35+
36+
See the [Minikube installation guide](https://kubernetes.io/docs/tasks/tools/install-minikube/)
37+
for details on installation, whilst paying attention to the following [Setup](#setting-up-minikube)
38+
section of this document. Also pay attention to the following
39+
[Nested Virtualization](#checking-for-nested-virtualization) section of this document, as
40+
Kata Containers will not function under Minikube if this is not enabled on your system.
41+
42+
## Setting up Minikube
43+
44+
To enable Kata Containers under Minikube, you need to add a few configuration options to the
45+
default Minikube setup. This is easily accomplished, as Minikube supports them on the setup commandline.
46+
47+
Here are the features, and why you need them:
48+
49+
| what | why |
50+
| ---- | --- |
51+
| `--vm-driver kvm2` | The host VM driver |
52+
| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb |
53+
| `--network-plugin=cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |
54+
| `--enable-default-cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |
55+
| `--container-runtime=cri-o` | Using CRI-O for Kata |
56+
| `--bootstrapper=kubeadm` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |
57+
58+
> **Note:** This example deploys Kata Containers using CRI-O. The same procedure should also work
59+
> for a `containerd` based Kubernetes installation, but is not covered in this document.
60+
61+
> **Note:** Prior to Minikube/Kubernetes v1.14, the beta `RuntimeClass` feature also needed enabling with
62+
> the following. Recent Minikube/Kubernetes installations should no longer require this setting.
63+
>
64+
> | what | why |
65+
> | ---- | --- |
66+
> | `--feature-gates=RuntimeClass=true` | Kata needs to use the RuntimeClass Kubernetes feature |
67+
68+
The full command is therefore:
69+
70+
```sh
71+
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
72+
```
73+
74+
> **Note:** For Kata Containers >v1.6.2, the now default `tcfilter` networking of Kata Containers
75+
> did not work for Minikube versions <v1.1.1. Please ensure you use Minikube version >=v1.1.1.
76+
77+
## Checking for nested virtualization
78+
79+
For Kata Containers to work under a Minikube VM, your host system must support
80+
nested virtualization. If you are using a Linux system utilising Intel VT-x
81+
and the `kvm_intel` driver, you can perform the following check:
82+
83+
```sh
84+
$ cat /sys/module/kvm_intel/parameters/nested
85+
Y
86+
```
87+
88+
If your system does not report `Y` from the `nested` parameter, then details on how
89+
to enable nested virtualisation can be found on the
90+
[KVM Nested Guests page](https://www.linux-kvm.org/page/Nested_Guests)
91+
92+
## Check Minikube is running
93+
94+
Before you install Kata Containers, perform a quick check that your Minikube is
95+
operating. On your guest:
96+
97+
98+
```sh
99+
$ kubectl get nodes
100+
NAME STATUS ROLES AGE VERSION
101+
minikube Ready master 78m v1.13.4
102+
```
103+
104+
Check you have virtualization enabled inside your Minikube:
105+
106+
```sh
107+
# No color option, as minikube is running busybox
108+
$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"
109+
110+
# if you get a long line of output here, you have it enabled!
111+
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti retpoline rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip
112+
113+
# A failed result means it is not enabled:
114+
ssh: Process exited with status 1
115+
```
116+
117+
## Installing Kata Containers
118+
119+
You can now install the Kata Containers runtime components. You will need a local copy of some Kata
120+
Containers components to help with this, and then use the host `kubectl` (that Minikube has already
121+
configured for you) to deploy them:
122+
123+
```sh
124+
$ git clone https://github.com/kata-containers/packaging.git
125+
$ cd packaging/kata-deploy
126+
$ kubectl apply -f kata-rbac.yaml
127+
$ kubectl apply -f kata-deploy.yaml
128+
```
129+
130+
This installs the Kata Containers components into `/opt/kata` inside the Minikube node. It can take
131+
a few minutes for the operation to complete. You can check the installation has worked by checking
132+
the status of the `kata-deploy` pod. This can be accomplished by running the following:
133+
134+
```sh
135+
$ podname=$(kubectl -n kube-system get pods -o=name | fgrep kata-deploy | sed 's?pod/??')
136+
# If 'sleep infinity' is running, the pod has completed its work.
137+
$ kubectl -n kube-system exec ${podname} -- ps -ef | fgrep infinity
138+
root 49 1 0 10:55 ? 00:00:00 sleep infinity
139+
```
140+
141+
> *NOTE:* This check only works for single node clusters, which is the default for Minikube.
142+
> For multi-node clusters, the check would need to be adapted to check `kata-deploy` had
143+
> completed on all nodes.
144+
145+
## Enabling Kata Containers
146+
147+
> **Note:** This step is only required for Minikube/Kubernetes versions <= 1.13. Since version
148+
> v1.14, the `RuntimeClass` is enabled by default. Performing this step on Kubernetes > v1.14 is
149+
> however benign.
150+
151+
Now the Kata Containers components are installed in the Minikube node, you need to configure
152+
Kubernetes `RuntimeClass` to know when to use Kata Containers to run a pod.
153+
154+
```sh
155+
$ curl https://raw.githubusercontent.com/kubernetes/node-api/master/manifests/runtimeclass_crd.yaml > runtimeclass_crd.yaml
156+
$ kubectl apply -f runtimeclass_crd.yaml
157+
```
158+
159+
### Register the runtime
160+
161+
Now register the `kata qemu` runtime with that class:
162+
163+
```sh
164+
$ cd packaging/kata-deploy/k8s-1.14
165+
$ kubectl apply -f kata-qemu-runtimeClass.yaml
166+
runtimeclass.node.k8s.io/kata created
167+
runtimeclass.node.k8s.io/kata-qemu created
168+
```
169+
170+
Kata Containers should now be installed and enabled in the Minikube cluster.
171+
172+
## Testing Kata Containers
173+
174+
Launch a container that is defined to run on Kata Containers. The enabling is configured by
175+
the following lines in the yaml file:
176+
177+
```yaml
178+
spec:
179+
runtimeClassName: kata-qemu
180+
```
181+
182+
Perform the following action to launch a Kata Containers based pod:
183+
184+
```sh
185+
$ cd packaging/kata-deploy/examples
186+
$ kubectl apply -f test-deploy-kata-qemu.yaml
187+
```
188+
189+
An Apache PHP pod should have been deployed with the Kata Containers runtime. Wait a few moments
190+
to check it is running:
191+
192+
```sh
193+
$ kubectl get pods
194+
NAME READY STATUS RESTARTS AGE
195+
php-apache-kata-qemu-bc4c55994-p6f6k 1/1 Running 0 1m
196+
```
197+
198+
There are a couple of ways to verify it is running with Kata Containers. Nominally, it
199+
should be hard to tell - the idea of Kata Containers is that your pod will run inside a VM,
200+
but look and feel just like it would as a normal software pod. In
201+
theory, you should not be able to tell your pod is running as a Kata Containers container.
202+
Careful examination can verify your pod is in fact a Kata Containers pod.
203+
204+
First, look on the node for a `qemu` running, and which version of the kernel the
205+
node is running:
206+
207+
```sh
208+
# You should see a VERY long line of qemu here - showing that you are indeed running
209+
# a qemu VM on the Minikube node - that is the VM that contains your pod.
210+
$ minikube ssh -- pgrep -a qemu
211+
212+
# For use in a moment, see what kernel is running on the node itself.
213+
$ minikube ssh -- uname -a
214+
Linux minikube 4.15.0 #1 SMP Wed Mar 6 23:18:58 UTC 2019 x86_64 GNU/Linux
215+
```
216+
217+
You should have seen a QEMU process running, indicating that your pod was indeed
218+
running inside a Kata Containers VM.
219+
220+
Another way to verify Kata Containers is running is to look in the container itself and check
221+
which kernel is running there. For a normal software container you will be running
222+
the same kernel as the node. For a Kata Container you will be running a Kata Containers kernel
223+
inside the Kata Containers VM.
224+
225+
```sh
226+
$ podname=$(kubectl get pods -o=name | fgrep php-apache-kata-qemu | sed 's?pod/??')
227+
$ kubectl exec ${podname} -- uname -a
228+
Linux php-apache-kata-qemu-7d4647498f-5cl9q 4.19.28 #1 SMP Wed Jun 5 14:32:34 PDT 2019 x86_64 GNU/Linux
229+
```
230+
231+
There you should see, the node is running kernel 4.15, but the container running under Kata Containers
232+
is running (in this instance) a 4.19 kernel.
233+
234+
## Wrapping up
235+
236+
This guide should have shown a relatively easy way to get a Minikube up with Kata Containers installed.
237+
Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup.
238+
It will have limitations, but, as a first introduction to Kata Containers, and how to install it under Kubernetes,
239+
it should suffice for initial learning and experimentation.
240+

0 commit comments

Comments
 (0)