|
| 1 | +# Conformance Testing in Kubernetes |
| 2 | + |
| 3 | +The Kubernetes conformance test suite is a set of testcases, currently a |
| 4 | +subset of the integration/e2e tests, that the Architecture SIG has approved |
| 5 | +to define the core set of interoperable features that all Kubernetes |
| 6 | +deployments must support. |
| 7 | + |
| 8 | +Contributors must write and submit e2e tests first (approved by owning Sigs). |
| 9 | +Once the new tests prove to be stable in CI runs, later create a follow up PR |
| 10 | +to add the test to conformance. This approach also decouples the development |
| 11 | +of useful tests from their promotion to conformance. |
| 12 | + |
| 13 | +A conformance test verifies the expected functionality works as a user might encounter it in the wild, |
| 14 | +and tests should begin by covering the most important and visible aspects of the function. |
| 15 | + |
| 16 | +Conformance tests in the default profile should cover features that are: |
| 17 | + |
| 18 | +- GA (not alpha or beta APIs, nor deprecated features) |
| 19 | +- portable (not dependent on provider-specific capabilities or on the public internet) |
| 20 | +- not an optional feature (e.g. not policy enforcement) |
| 21 | +- non-privileged (neither root on nodes, network, nor cluster) |
| 22 | +- should be stable and run consistently |
| 23 | +- cannot skip providers (there should be no Skip like directives added to the test), |
| 24 | +especially in the Nucleus or Application layers as described |
| 25 | +[here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md). |
| 26 | +- should work with default settings for all configuration parameters |
| 27 | +(example: the default list of admission plugins should not have to be tweaked for passing conformance). |
| 28 | + |
| 29 | +For each Kubernetes release, a Conformance Document will be generated |
| 30 | +that lists all of the tests that comprise the conformance test suite, along |
| 31 | +with the formal specification of each test. For example conformance document for |
| 32 | +1.9 can be found [here](https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md). |
| 33 | +This document will help people understand what features are being tested without having to look through |
| 34 | +the testcase's code directly. |
| 35 | + |
| 36 | +As each new release of Kubernetes provides new functionality, the subset of |
| 37 | +tests necessary to demonstrate conformance grows with each release. Conformance |
| 38 | +is thus considered versioned, with the same backwards compatibility guarantees |
| 39 | +as laid out in [our versioning policy](/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew). |
| 40 | +Conformance tests for a given version should be run off of the release branch |
| 41 | +that corresponds to that version. Thus `v1.2` conformance tests would be run |
| 42 | +from the head of the `release-1.2` branch. eg: |
| 43 | + |
| 44 | + - A v1.3 development cluster should pass v1.1, v1.2 conformance tests |
| 45 | + |
| 46 | + - A v1.2 cluster should pass v1.1, v1.2 conformance tests |
| 47 | + |
| 48 | + - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 |
| 49 | +conformance tests |
| 50 | + |
| 51 | +Conformance tests are designed to be run with no cloud provider configured. |
| 52 | +Conformance tests can be run against clusters that have not been created with |
| 53 | +`hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and |
| 54 | +credentials. |
| 55 | + |
| 56 | +```sh |
| 57 | +# setup for conformance tests |
| 58 | +export KUBECONFIG=/path/to/kubeconfig |
| 59 | +export KUBERNETES_CONFORMANCE_TEST=y |
| 60 | + |
| 61 | +# run all conformance tests |
| 62 | +go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" |
| 63 | + |
| 64 | +# run all parallel-safe conformance tests in parallel |
| 65 | +GINKGO_PARALLEL=y go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" |
| 66 | + |
| 67 | +# ... and finish up with remaining tests in serial |
| 68 | +go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" |
| 69 | +``` |
| 70 | + |
| 71 | +### Defining Conformance Subset |
| 72 | + |
| 73 | +It is impossible to define the entire space of Conformance tests without knowing |
| 74 | +the future, so instead, we define the compliment of conformance tests, below |
| 75 | +(`Please update this with companion PRs as necessary`): |
| 76 | + |
| 77 | + - A conformance test cannot test cloud provider specific features (i.e. GCE |
| 78 | +monitoring, S3 Bucketing, ...) |
| 79 | + |
| 80 | + - A conformance test cannot rely on any particular non-standard file system |
| 81 | +permissions granted to containers or users (i.e. sharing writable host /tmp with |
| 82 | +a container) |
| 83 | + |
| 84 | + - A conformance test cannot rely on any binaries that are not required for the |
| 85 | +linux kernel or for a kubelet to run (i.e. git) |
| 86 | + |
| 87 | + - A conformance test cannot test a feature which obviously cannot be supported |
| 88 | +on a broad range of platforms (i.e. testing of multiple disk mounts, GPUs, high |
| 89 | +density) |
| 90 | + |
| 91 | +## Adding New Tests |
| 92 | + |
| 93 | +To promote a testcase to the conformance test suite, the following |
| 94 | +steps must be taken: |
| 95 | +- as a prerequisite, the test case is already part of e2e and is not flaky |
| 96 | +- the testcase must use the `framework.ConformanceIt()` function rather |
| 97 | + than the `framework.It()` function |
| 98 | +- the testcase must include a comment immediately before the |
| 99 | + `framework.ConformanceIt()` call that includes all of the required |
| 100 | + metadata about the test (see the [Test Metadata](#test-metadata) section) |
| 101 | +- use "Promote xxx e2e test to Conformance" as template of your PR title |
| 102 | +- tag your PR with "/area conformance" label |
| 103 | +- send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" |
| 104 | +also CC the relevant Sig and Sig-Architecture |
| 105 | + |
| 106 | + |
| 107 | +## Test Metadata |
| 108 | + |
| 109 | +Each conformance test must include the following piece of metadata |
| 110 | +within its associated comment: |
| 111 | + |
| 112 | +- `Release`: indicates the Kubernetes release that the test was added to the |
| 113 | + conformance test suite. If the test was modified in subsequent releases |
| 114 | + then those releases should be included as well (comma separated) |
| 115 | +- `Testname`: a human readable short name of the test |
| 116 | +- `Description`: a detailed description of the test. This field must describe |
| 117 | + the required behaviour of the Kubernetes components being tested using |
| 118 | + [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field |
| 119 | + is meant to be a "specification" of the tested Kubernetes features, as |
| 120 | + such, it must be detailed enough so that readers can fully understand |
| 121 | + the aspects of Kubernetes that are being tested without having to read |
| 122 | + the test's code directly. Additionally, this test should provide a clear |
| 123 | + distinction between the parts of the test that are there for the purpose |
| 124 | + of validating Kubernetes rather than simply infrastructure logic that |
| 125 | + is necessary to setup, or clean up, the test. |
| 126 | + |
| 127 | +## Sample Conformance Test |
| 128 | + |
| 129 | +The following snippet of code shows a sample conformance test's metadata: |
| 130 | + |
| 131 | +``` |
| 132 | +/* |
| 133 | + Release : v1.9 |
| 134 | + Testname: Kubelet: log output |
| 135 | + Description: By default the stdout and stderr from the process being |
| 136 | + executed in a pod MUST be sent to the pod's logs. |
| 137 | +*/ |
| 138 | +framework.ConformanceIt("it should print the output to logs", func() { |
| 139 | + ... |
| 140 | +}) |
| 141 | +``` |
| 142 | + |
| 143 | +The corresponding portion of the Conformance Document would then look |
| 144 | +like this: |
| 145 | + |
| 146 | +> |
| 147 | +> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) |
| 148 | +> |
| 149 | +> Release : v1.9 |
| 150 | +> |
| 151 | +> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. |
| 152 | +
|
| 153 | +## Reporting Conformance Test Results |
| 154 | + |
| 155 | +Conformance test results, by provider and releases, can be viewed in the |
| 156 | +federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conformance-all). |
| 157 | +If you wish to contribute conformance test results for your provider, |
| 158 | +please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#). |
| 159 | + |
0 commit comments