|
| 1 | +# Conformance Test Guidelines |
| 2 | + |
| 3 | +The Kubernetes conformance test suite is a set of testcases, currently a |
| 4 | +subset of the integration/e2e tests, that the Architecture SIG has approved |
| 5 | +to define the core set of interoperable features that all Kubernetes |
| 6 | +deployments must support. |
| 7 | + |
| 8 | +Contributors must write and submit e2e tests first (approved by owning Sigs). |
| 9 | +Once the new tests prove to be stable in CI runs, later create a follow up PR |
| 10 | +to add the test to conformance. This approach also decouples the production |
| 11 | +of useful tests from their promotion to conformance. |
| 12 | + |
| 13 | +A conformance test verifies the expected functionality works as a user might encounter it in the wild, |
| 14 | +and tests should begin by covering the most important and visible aspects of the function. |
| 15 | + |
| 16 | +Conformance tests in the default profile should cover features that are: |
| 17 | + |
| 18 | +- GA (not alpha or beta APIs, nor deprecated features) |
| 19 | +- portable (not dependent on provider-specific capabilities or on the public internet) |
| 20 | +- not an optional feature (e.g. not policy enforcement) |
| 21 | +- non-privileged (neither root on nodes, network, nor cluster) |
| 22 | +- cannot be flaky |
| 23 | +- cannot skip providers (there should be no Skip like directives added to the test), |
| 24 | +especially in the Nucleus or Application layers as described |
| 25 | +[here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md). |
| 26 | +- should work with default settings for all configuration parameters |
| 27 | +(example: the default list of admission plugins should not have to be tweaked for passing conformance). |
| 28 | + |
| 29 | +For each Kubernetes release, a Conformance Document will be generated |
| 30 | +that lists all of the tests that comprise the conformance test suite, along |
| 31 | +with the formal specification of each test. For example conformance document for |
| 32 | +1.9 can be found [here](https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md). |
| 33 | +This document will help people understand what features are being tested without having to look through |
| 34 | +the testcase's code directly. |
| 35 | + |
| 36 | +## Adding New Tests |
| 37 | + |
| 38 | +To promote a testcase to the conformance test suite, the following |
| 39 | +steps must be taken: |
| 40 | +- as a prerequisite, the test case is already part of e2e and is not flaky |
| 41 | +- the testcase must use the `framework.ConformanceIt()` function rather |
| 42 | + than the `framework.It()` function |
| 43 | +- the testcase must include a comment immediately before the |
| 44 | + `framework.ConformanceIt()` call that includes all of the required |
| 45 | + metadata about the test (see the [Test Metadata](#test-metadata) section) |
| 46 | +- use "Promote xxx e2e test to Conformance" as template of your PR title |
| 47 | +- tag your PR with "/area conformance" label |
| 48 | +- send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" |
| 49 | +also CC the relevant Sig and Sig-Architecture |
| 50 | + |
| 51 | + |
| 52 | +## Test Metadata |
| 53 | + |
| 54 | +Each conformance test must include the following piece of metadata |
| 55 | +within its associated comment: |
| 56 | + |
| 57 | +- `Release`: indicates the Kubernetes release that the test was added to the |
| 58 | + conformance test suite. If the test was modified in subsequent releases |
| 59 | + then those releases should be included as well (comma separated) |
| 60 | +- `Testname`: a human readable short name of the test |
| 61 | +- `Description`: a detailed description of the test. This field must describe |
| 62 | + the required behaviour of the Kubernetes components being tested using |
| 63 | + [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field |
| 64 | + is meant to be a "specification" of the tested Kubernetes features, as |
| 65 | + such, it must be detailed enough so that readers can fully understand |
| 66 | + the aspects of Kubernetes that are being tested without having to read |
| 67 | + the test's code directly. Additionally, this test should provide a clear |
| 68 | + distinction between the parts of the test that are there for the purpose |
| 69 | + of validating Kubernetes rather than simply infrastructure logic that |
| 70 | + is necessary to setup, or clean up, the test. |
| 71 | + |
| 72 | +## Sample Conformance Test |
| 73 | + |
| 74 | +The following snippet of code shows a sample conformance test's metadata: |
| 75 | + |
| 76 | +``` |
| 77 | +/* |
| 78 | + Release : v1.9 |
| 79 | + Testname: Kubelet: log output |
| 80 | + Description: By default the stdout and stderr from the process being |
| 81 | + executed in a pod MUST be sent to the pod's logs. |
| 82 | +*/ |
| 83 | +framework.ConformanceIt("it should print the output to logs", func() { |
| 84 | + ... |
| 85 | +}) |
| 86 | +``` |
| 87 | + |
| 88 | +The corresponding portion of the Conformance Document would then look |
| 89 | +like this: |
| 90 | + |
| 91 | +> |
| 92 | +> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) |
| 93 | +> |
| 94 | +> Release : v1.9 |
| 95 | +> |
| 96 | +> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. |
| 97 | +
|
| 98 | + |
| 99 | +Conformance test results, by provider and releases, can be viewed in the |
| 100 | +federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conformance-all). |
| 101 | +If you wish to contribute conformance test results for your provider, |
| 102 | +please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#). |
| 103 | + |
0 commit comments