Document running E2E tests locally

Author: mdbooth

docs/book/src/development/development.md

@@ -42,3 +42,103 @@ After generating `infrastructure-components.yaml`, replace the `us.gcr.io/k8s-ar
 ## Developing with Tilt
 
 We have support for using [Tilt](https://tilt.dev/) for rapid iterative development. Please visit the [Cluster API documentation on Tilt](https://master.cluster-api.sigs.k8s.io/developer/tilt.html) for information on how to set up your development environment. 
+
+## Running E2E tests locally
+
+You can run the E2E tests locally with:
+
+```bash
+make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=mycloud
+```
+
+where `mycloud` is an entry in `clouds.yaml`.
+
+The E2E tests:
+* Build a CAPO image from the local working directory
+* Create a kind cluster locally
+* Deploy downloaded CAPI, and locally-build CAPO to kind
+* Create an e2e namespace per-test on the kind cluster
+* Deploy cluster templates to the test namespace
+* Create test clusters on the target OpenStack
+
+### Support for clouds using SSL
+
+If your cloud requires a cacert you must also pass this to make via `OPENSTACK_CLOUD_CACERT_B64`, i.e.:
+
+```bash
+make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
+              OPENSTACK_CLOUD_CACERT_B64=$(base64 -w /path/to/mycloud-ca.crt)
+```
+
+CAPO deployed in the local kind cluster will automatically pick up a `cacert` defined in your `clouds.yaml` so you will see servers created in OpenStack without specifying `OPENSTACK_CLOUD_CACERT_B64`. However, the cacert won't be deployed to those servers, so kubelet will fail to start.
+
+### Support for clouds with multiple external networks
+
+If your cloud contains only a single external network CAPO will automatically select that network for use by a deployed cluster. However, if there are multiple external networks CAPO will log an error and fail to create any machines. In this case you must pass the id of an external network to use explicitly with `OPENSTACK_EXTERNAL_NETWORK_ID`, i.e.:
+
+```bash
+make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
+              OPENSTACK_EXTERNAL_NETWORK_ID=27635f93-583d-454e-9c6d-3d305e7f8a22
+```
+
+`OPENSTACK_EXTERNAL_NETWORK_ID` must be specified as a uuid. Specifying by name is not supported.
+
+You can list available external networks with:
+
+```bash
+$ openstack network list --external
++--------------------------------------+----------+--------------------------------------+
+| ID                                   | Name     | Subnets                              |
++--------------------------------------+----------+--------------------------------------+
+| 27635f93-583d-454e-9c6d-3d305e7f8a22 | external | be64cd07-f8b7-4705-8446-26b19eab3914 |
+| cf2e83dc-545d-490f-9f9c-4e90927546f2 | hostonly | ec95befe-72f4-4af6-a263-2aec081f47d3 |
++--------------------------------------+----------+--------------------------------------+
+```
+
+### OpenStack prerequisites
+
+The file `test/e2e/data/e2e_conf.yaml` and the test templates under `test/e2e/data/infrastructure-openstack` reference several OpenStack resources which must exist before running the test:
+
+* Glance images
+  * `cirros-0.5.1-x86_64-disk`
+    * Download from https://docs.openstack.org/image-guide/obtain-images.html
+  * `ubuntu-2004-kube-v1.18.15`
+    * Download from https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2021-03-27/ubuntu-2004-kube-v1.18.15.qcow2
+    * Or generate using the `images/capi` directory from https://github.com/kubernetes-sigs/image-builder
+* Flavors
+  * `m1.medium`: used by control plane
+  * `m1.small`: used by workers
+  * `m1.tiny`: used by bastion
+
+## Running E2E tests using rootless podman
+
+You can use unprivileged podman to:
+* Build the CAPO image
+* Deploy the kind cluster
+
+To do this you need to configure the host appropriately and pass `PODMAN=1` to make, i.e.:
+
+```bash
+make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
+              PODMAN=1
+```
+
+### Host configuration
+
+Firstly, you must be using kernel >=5.11. Fedora 34 is the oldest distribution with this kernel.
+
+You *MUST* configure systemd and iptables as described in https://kind.sigs.k8s.io/docs/user/rootless/. There is no need to configure cgroups v2 as Fedora uses this by default.
+
+You must install the `podman-docker` package to emulate the docker cli tool. However, this is not sufficient on its own as described below.
+
+### Running podman system service to emulate docker daemon
+
+While kind itself supports podman, the cluster-api test framework does not. This framework is used by the CAPO tests to push test images into the kind cluster. Unfortunately the cluster-api test framework explicitly connects to a running docker daemon, so cli emulation is not sufficient for compatibility. This issue is tracked in https://github.com/kubernetes-sigs/cluster-api/issues/5146, and the following workaround can be ignored when this is resolved.
+
+podman includes a 'system service' which emulates docker. For the tests to work, this service must be running and listening on a unix socket at `/var/run/docker.sock`. You can achieve this with:
+
+```bash
+$ podman system service -t 0 &
+$ sudo rm /var/run/docker.sock
+$ sudo ln -s /run/user/$(id -u)/podman/podman.sock /var/run/docker.sock
+```