Skip to content

Commit cee0ad7

Browse files
brahmaroutubrahmaroutu
committed
Adding Conformance Guidelines document.
1 parent bc690fa commit cee0ad7

File tree

2 files changed

+105
-0
lines changed

2 files changed

+105
-0
lines changed

‎contributors/devel/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ Guide](http://kubernetes.io/docs/admin/).
4343
* **Document Conventions** ([how-to-doc.md](how-to-doc.md))
4444
Document style advice for contributors.
4545

46+
* **Conformance Tests Guidelines** ([conformance-guidelines.md](conformance-guidelines.md))
47+
Documenting Conformance tests.
48+
4649
* **Running a cluster locally** ([running-locally.md](running-locally.md)):
4750
A fast and lightweight local cluster deployment for development.
4851

‎contributors/devel/conformance-guidelines.md

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Conformance Test Guidelines
2+
3+
The Kubernetes conformance test suite is a set of testcases, currently a
4+
subset of the integration/e2e tests, that the Architecture SIG has approved
5+
to define the core set of interoperable features that all Kubernetes
6+
deployments must support.
7+
8+
Contributors must write and submit e2e tests first (approved by owning Sigs).
9+
Once the new tests prove to be stable in CI runs, later create a follow up PR
10+
to add the test to conformance. This approach also decouples the production
11+
of useful tests from their promotion to conformance.
12+
13+
A conformance test verifies the expected functionality works as a user might encounter it in the wild,
14+
and tests should begin by covering the most important and visible aspects of the function.
15+
16+
Conformance tests in the default profile should cover features that are:
17+
18+
- GA (not alpha or beta APIs, nor deprecated features)
19+
- portable (not dependent on provider-specific capabilities or on the public internet)
20+
- not an optional feature (e.g. not policy enforcement)
21+
- non-privileged (neither root on nodes, network, nor cluster)
22+
- cannot be flaky
23+
- cannot skip providers (there should be no Skip like directives added to the test),
24+
especially in the Nucleus or Application layers as described
25+
[here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md).
26+
- should work with default settings for all configuration parameters
27+
(example: the default list of admission plugins should not have to be tweaked for passing conformance).
28+
29+
For each Kubernetes release, a Conformance Document will be generated
30+
that lists all of the tests that comprise the conformance test suite, along
31+
with the formal specification of each test. This document will help people
32+
understand what features are being tested without having to look through
33+
the testcase's code directly.
34+
35+
## Adding New Tests
36+
37+
To include a testcase in the conformance test suite, the following
38+
steps must be taken:
39+
- the testcase must use the `framework.ConformanceIt()` function rather
40+
than the `framework.It()` function
41+
- the testcase must include a comment immediately before the
42+
`framework.ConformanceIt()` call that includes all of the required
43+
metadata about the test (see the [Test Metadata](#test-metadata) section)
44+
- use "Promote xxx e2e test to Conformance" as template of your PR title
45+
- tag your PR with "/area conformance" label
46+
- send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews"
47+
also CC the relevant Sig and Sig-Architecture
48+
49+
The PR associated with this change will be automatically be blocked until the
50+
Architecture SIG has reviewed and approved the change. The mechanism by which
51+
the Architecture SIG is notified of the desired change is still TBD.
52+
53+
## Test Metadata
54+
55+
Each conformance test must include the following piece of metadata
56+
within its associated comment:
57+
58+
- `Release`: indicates the Kubernetes release that the test was added to the
59+
conformance test suite. If the test was modified in subsequent releases
60+
then those releases should be included as well (comma separated)
61+
- `Testname`: a human readable short name of the test
62+
- `Description`: a detailed description of the test. This field must describe
63+
the required behaviour of the Kubernetes components being tested using
64+
[RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field
65+
is meant to be a "specification" of the tested Kubernetes features, as
66+
such, it must be detailed enough so that readers can fully understand
67+
the aspects of Kubernetes that are being tested without having to read
68+
the test's code directly. Additionally, this test should provide a clear
69+
distinction between the parts of the test that are there for the purpose
70+
of validating Kubernetes rather than simply infrastructure logic that
71+
is necessary to setup, or clean up, the test.
72+
73+
## Sample Conformance Test
74+
75+
The following snippet of code shows a sample conformance test's metadata:
76+
77+
```
78+
/*
79+
Release : v1.9
80+
Testname: Kubelet: log output
81+
Description: By default the stdout and stderr from the process being
82+
executed in a pod MUST be sent to the pod's logs.
83+
*/
84+
framework.ConformanceIt("it should print the output to logs", func() {
85+
...
86+
})
87+
```
88+
89+
The corresponding portion of the Conformance Document would then look
90+
like this:
91+
92+
>
93+
> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47)
94+
>
95+
> Release : v1.9
96+
>
97+
> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs.
98+
99+
100+
Conformance test results can also be viewed from the testgrid, details can be found [here](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#).
101+
102+

0 commit comments

Comments
 (0)