docs: Clean up k8s with cri-containerd howto

This PR is based on #124 but has been reworked and updated to take into
account review feedback and extra cleanups to bring this howto in line
with the latest documentation requirements.

Fixes #127.

Contributions-from: T. Nichole Williams <[email protected]>
Signed-off-by: James O. D. Hunt <[email protected]>

Author: jodh-intel

how-to/how-to-use-k8s-with-cri-containerd-and-kata.md

@@ -1,64 +1,82 @@
 # How to use Kata Containers and CRI (containerd plugin) with Kubernetes
 
-This document describes how to set up a single-machine Kubernetes cluster. 
-The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads.
+* [Requirements](#requirements)
+* [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled)
+* [Install Kata Containers](#install-kata-containers)
+* [Install Kubernetes](#install-kubernetes)
+* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
+    * [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime)
+* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd)
+* [Configure proxy - OPTIONAL](#configure-proxy---optional)
+* [Start Kubernetes](#start-kubernetes)
+* [Install a Pod Network](#install-a-pod-network)
+* [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node)
+* [Create an unstrusted pod using Kata Containers](#create-an-unstrusted-pod-using-kata-containers)
+* [Delete created pod](#delete-created-pod)
+
+This document describes how to set up a single-machine Kubernetes (k8s) cluster.
+
+The Kubernetes cluster will use the
+[CRI containerd plugin](https://github.com/containerd/cri) and
+[Kata Containers](https://katacontainers.io) to launch untrusted workloads.
+
+## Requirements
 
-## Requirements 
 - Kubernetes, kubelet, kubeadm
 - cri-containerd
 - Kata Containers
 
-For information about the supported version of these components see 
-Kata Containers [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) file.
+**Note:** For information about the supported versions of these components,
+see the  Kata Containers
+[versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml)
+file.
 
-## Install containerd(with CRI plugin enabled)
+## Install containerd with CRI plugin enabled
 
-Follow the instructions from [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md)
+- Follow the instructions from the
+  [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).
 
-<!---
-```bash
-# Check if containerd is installed
-$ command -v containerd
-```
---->
-
-## Install Kata Containers 
+- Check if `containerd` is now available
+  ```bash
+  $ command -v containerd
+  ```
 
-Follow the instructions to [install Kata](https://github.com/kata-containers/documentation/blob/master/install/README.md).
+## Install Kata Containers
 
-<!---
-```bash
-# Check if kata-runtime is installed
-$ command -v kata-runtime
-# Check kata is well configured
-$ kata-runtime kata-env
-```
---->
+Follow the instructions to
+[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
 
 ## Install Kubernetes
-Install Kubernetes in your host.  See kubeadm [installation](https://kubernetes.io/docs/tasks/tools/install-kubeadm/)
-<!---
-```bash
-# Check if kubadm is installed
-$ command -v kubeadm
-```
---->
 
-### Configure containerd to use Kata Containers
+- Follow the instructions for
+  [kubeadm installation](https://kubernetes.io/docs/setup/independent/install-kubeadm/).
+
+- Check `kubeadm` is now available
+
+  ```bash
+  $ command -v kubeadm
+  ```
+
+## Configure containerd to use Kata Containers
+
+The CRI `containerd` plugin supports configuration for two runtime types.
 
-The CRI containerd plugin support configuration for two runtime types.
+- **Default runtime:**
 
-- Default runtime: A runtime that is used by default to run workloads.
-- Untrusted workload runtime: A runtime that will be used run untrusted workloads.
+  A runtime that is used by default to run workloads.
 
-#### Define the Kata runtime as `untrusted_workload_runtime` 
+- **Untrusted workload runtime:**
 
-Configure the Kata runtime for untrusted workload with the [config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md)
-`plugins.cri.containerd.untrusted_workload_runtime`.
+  A runtime that will be used to run untrusted workloads. This is appropriate
+  for workloads that require a higher degree of security isolation.
+
+#### Define the Kata runtime as the untrusted workload runtime
+
+Configure `containerd` to use the Kata runtime to run untrusted workloads by
+setting the `plugins.cri.containerd.untrusted_workload_runtime`
+[config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):
 
-Unless configured otherwise, the default runtime is set to `runc`.
 ```bash
-# Configure containerd to use Kata as untrusted_workload_runtime
 $ sudo mkdir -p /etc/containerd/
 $ cat << EOT | sudo tee /etc/containerd/config.toml
 [plugins]
@@ -69,28 +87,33 @@ $ cat << EOT | sudo tee /etc/containerd/config.toml
 EOT
 ```
 
-### Configure Kubelet to use containerd
+**Note:** Unless configured otherwise, the default runtime is set to `runc`.
 
-In order to allow kubelet use containerd (using CRI interface) configure the service to
-point to containerd socket.
+## Configure Kubelet to use containerd
 
-```bash
-# Configure k8s to use containerd
-$ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
-$ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
-[Service]                                                 
-Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
-EOF
-$ sudo systemctl daemon-reload
-```
+In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket.
 
-### Optional: Configure proxy
+- Configure Kubernetes to use `containerd`
 
-If you are behind a proxy this script will configure your proxy for docker
-kubelet and containerd.
+  ```bash
+  $ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
+  $ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
+  [Service]                                                 
+  Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
+  EOF
+  ```
+
+- Inform systemd about the new configuration
+
+  ```bash
+  $ sudo systemctl daemon-reload
+  ```
+
+## Configure proxy - OPTIONAL
+
+If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd:
 
 ```bash
-# Set proxys
 $ services=(
 'kubelet'
 'containerd'
@@ -99,115 +122,134 @@ $ services=(
 
 $ for s in "${services[@]}"; do 
 
-	service_dir="/etc/systemd/system/${s}.service.d/"
-	sudo mkdir -p ${service_dir}
+    service_dir="/etc/systemd/system/${s}.service.d/"
+    sudo mkdir -p ${service_dir}
 
-	cat << EOT | sudo tee "${service_dir}/proxy.conf"
+    cat << EOT | sudo tee "${service_dir}/proxy.conf"
 [Service]
 Environment="HTTP_PROXY=${http_proxy}"
 Environment="HTTPS_PROXY=${https_proxy}"
 Environment="NO_PROXY=${no_proxy}"
 EOT
 done
+
 $ sudo systemctl daemon-reload
 ```
 
-###  Start Kubernetes with kubeadm 
+## Start Kubernetes
 
-```bash
-# Mark sure containerd is up and running
-$ sudo systemctl restart containerd
-$ sudo systemctl status containerd
+- Make sure `containerd` is up and running
 
-# Prevent docker iptables rules conflict with k8s pod communication
-$ sudo iptables -P FORWARD ACCEPT
+  ```bash
+  $ sudo systemctl restart containerd
+  $ sudo systemctl status containerd
+  ```
 
-# Start cluster using kubeadm
-$ sudo kubeadm init --skip-preflight-checks \
---cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
+- Prevent conflicts between `docker` iptables (packet filtering) rules and k8s pod communication
 
-$ export KUBECONFIG=/etc/kubernetes/admin.conf
+  ```bash
+  $ sudo iptables -P FORWARD ACCEPT
+  ```
 
-$ sudo -E kubectl get nodes
-$ sudo -E kubectl get pods
-```
+- Start cluster using `kubeadm`
 
-### Install a pod network
-Install a pod network plugin is needed to allow pods communicate with each other.
+  ```bash
+  $ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
 
-Install flannel plugging, by following the instructions in the section *Installing a pod network*
-from [Using kubeadm to Create a Cluster ](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
-guide.
+  $ export KUBECONFIG=/etc/kubernetes/admin.conf
 
-<!---
-```bash
-# Install a pod network using flannel
-# There is not a programmatic way to know last what flannel commit use
-# See https://github.com/coreos/flannel/issues/995
-$ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
-```
---->
+  $ sudo -E kubectl get nodes
+  $ sudo -E kubectl get pods
+  ```
 
+## Install a Pod Network
 
-```bash
-# wait for pod network
-$ timeout_dns=0
-$ until [ "$timeout_dns" -eq "420" ]; do
-	if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
-		break
-	fi
-	sleep 1s
-	((timeout_dns+=1))
- done
-
-# check pod network is running
-$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
-```
+A pod network plugin is needed to allow pods to communicate with each other.
 
-### Allow run pods in master node
+- Install the `flannel` plugin by following the
+  [Using kubeadm to Create a Cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions)
+  guide, starting from the **Installing a pod network** section.
 
-By default, the cluster will not schedule pods in the master node to allow that run:
+- Create a pod network using flannel
 
-```bash
-# allow master node run pods
-$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
-```
+  **Note:** There is not a programmatic way to know last what flannel commit use. See https://github.com/coreos/flannel/issues/995.
 
+  ```bash
+  $ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
+  ```
 
-### Create a unstrusted pod using Kata Containers
+- Wait for the pod network to become available
 
-By default, all pods are created with the default runtime configured in CRI containerd plugin.
-If a pod has the `io.kubernetes.cri.untrusted-workload annotation` set as
-`"true"`, the CRI plugin will run the pod with the Kata Containers runtime.
+  ```bash
+  # number of seconds to wait for pod network to become available
+  $ timeout_dns=420
 
-```bash
-# Create untrusted pod configuration
-$ cat << EOT | tee nginx-untrusted.yaml
-apiVersion: v1
-kind: Pod
-metadata:
-  name: nginx-untrusted
-  annotations:
-    io.kubernetes.cri.untrusted-workload: "true"
-spec:
-  containers:
-  - name: nginx
-    image: nginx
-    
-EOT
+  $ while [ "$timeout_dns" -gt 0 ]; do
+      if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
+          break
+      fi
+
+      sleep 1s
+      ((timeout_dns-=1))
+   done
+  ```
 
-# Create untrusted pod
-$ sudo -E kubectl apply -f nginx-untrusted.yaml
+- Check the pod network is running
+  ```bash
+  $ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
+  ```
 
-# Check pod is running
-$ sudo -E kubectl get pods
+## Allow pods to run in the master node
 
-# Check qemu is running
-$ ps aux | grep qemu
+By default, the cluster will not schedule pods in the master node. To enable master node scheduling:
+
+```bash
+$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
 ```
-### Delete created pod
+
+## Create an unstrusted pod using Kata Containers
+
+By default, all pods are created with the default runtime configured in CRI containerd plugin.
+
+If a pod has the `io.kubernetes.cri.untrusted-workload` annotation set to `"true"`, the CRI plugin runs the pod with the
+[Kata Containers runtime](https://github.com/kata-containers/runtime/blob/master/README.md).
+
+- Create an untrusted pod configuration
+
+  ```bash
+  $ cat << EOT | tee nginx-untrusted.yaml
+  apiVersion: v1
+  kind: Pod
+  metadata:
+    name: nginx-untrusted
+    annotations:
+      io.kubernetes.cri.untrusted-workload: "true"
+  spec:
+    containers:
+    - name: nginx
+      image: nginx
+      
+  EOT
+  ```
+
+- Create an untrusted pod
+  ```bash
+  $ sudo -E kubectl apply -f nginx-untrusted.yaml
+  ```
+
+- Check pod is running
+
+  ```bash
+  $ sudo -E kubectl get pods
+  ```
+
+- Check hypervisor is running
+  ```bash
+  $ ps aux | grep qemu
+  ```
+
+## Delete created pod
 
 ```bash
-# Delete pod
-$ sudo -E kubectl delete -f  nginx-untrusted.yaml
+$ sudo -E kubectl delete -f nginx-untrusted.yaml
 ```