-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Adding Conformance Guidelines document. #1867
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| # Conformance Testing in Kubernetes | ||
|
|
||
| The Kubernetes conformance test suite is a set of testcases, currently a | ||
| subset of the integration/e2e tests, that the Architecture SIG has approved | ||
| to define the core set of interoperable features that all Kubernetes | ||
| deployments must support. | ||
|
|
||
| Contributors must write and submit e2e tests first (approved by owning Sigs). | ||
| Once the new tests prove to be stable in CI runs, later create a follow up PR | ||
| to add the test to conformance. This approach also decouples the development | ||
| of useful tests from their promotion to conformance. | ||
|
|
||
| A conformance test verifies the expected functionality works as a user might encounter it in the wild, | ||
| and tests should begin by covering the most important and visible aspects of the function. | ||
|
|
||
| ### Conformance Test Requirements | ||
|
|
||
| A test is eligible for promotion to conformance if it meets the following requirements: | ||
|
|
||
| - testing GA feature (not alpha or beta APIs, nor deprecated features) | ||
| - must be portable (not dependent on provider-specific capabilities or on the public internet) | ||
| - cannot test a feature which obviously cannot be supported on a broad range of platforms | ||
| (i.e. testing of multiple disk mounts, GPUs, high density) | ||
| - cannot test an optional feature (e.g. not policy enforcement) | ||
| - should be non-privileged (neither root on nodes, network, nor cluster) | ||
| - cannot rely on any particular non-standard file system permissions granted to containers or users | ||
| (i.e. sharing writable host /tmp with a container) | ||
| - should be stable and run consistently | ||
| - cannot skip providers (there should be no Skip like directives added to the test), | ||
| especially in the Nucleus or Application layers as described | ||
| [here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md). | ||
| - cannot test cloud provider specific features (i.e. GCE monitoring, S3 Bucketing, ...) | ||
| - should work with default settings for all configuration parameters | ||
| (example: the default list of admission plugins should not have to be tweaked for passing conformance). | ||
| - cannot rely on any binaries that are not required for the | ||
| linux kernel or for a kubelet to run (i.e. git) | ||
|
|
||
| ### Conformance Test Version Skew Policy | ||
|
|
||
| As each new release of Kubernetes provides new functionality, the subset of | ||
| tests necessary to demonstrate conformance grows with each release. Conformance | ||
| is thus considered versioned, with the same backwards compatibility guarantees | ||
| as laid out in [our versioning policy](/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew). | ||
| Conformance tests for a given version should be run off of the release branch | ||
| that corresponds to that version. Thus `v1.2` conformance tests would be run | ||
| from the head of the `release-1.2` branch. eg: | ||
|
|
||
| - A v1.3 development cluster should pass v1.1, v1.2 conformance tests | ||
|
|
||
| - A v1.2 cluster should pass v1.1, v1.2 conformance tests | ||
|
|
||
| - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 | ||
| conformance tests | ||
|
|
||
|
|
||
| ### Running Conformance Tests | ||
|
|
||
| Conformance tests are designed to be run with no cloud provider configured. | ||
| Conformance tests can be run against clusters that have not been created with | ||
| `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and | ||
| credentials. | ||
|
|
||
| ```sh | ||
| # setup for conformance tests | ||
| export KUBECONFIG=/path/to/kubeconfig | ||
| export KUBERNETES_CONFORMANCE_TEST=y | ||
|
|
||
| # run all conformance tests | ||
| go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" | ||
|
|
||
| # run all parallel-safe conformance tests in parallel | ||
| GINKGO_PARALLEL=y go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" | ||
|
|
||
| # ... and finish up with remaining tests in serial | ||
| go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" | ||
| ``` | ||
|
|
||
| ### Kubernetes Conformance Document | ||
| For each Kubernetes release, a Conformance Document will be generated | ||
| that lists all of the tests that comprise the conformance test suite, along | ||
| with the formal specification of each test. For example conformance document for | ||
| 1.9 can be found [here](https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md). | ||
| This document will help people understand what features are being tested without having to look through | ||
| the testcase's code directly. | ||
|
|
||
|
|
||
| ## Adding New Tests | ||
|
|
||
| To promote a testcase to the conformance test suite, the following | ||
| steps must be taken: | ||
| - as a prerequisite, the test case is already part of e2e and is not flaky | ||
| - the testcase must use the `framework.ConformanceIt()` function rather | ||
| than the `framework.It()` function | ||
| - the testcase must include a comment immediately before the | ||
| `framework.ConformanceIt()` call that includes all of the required | ||
| metadata about the test (see the [Test Metadata](#test-metadata) section) | ||
| - use "Promote xxx e2e test to Conformance" as template of your PR title | ||
| - tag your PR with "/area conformance" label | ||
| - send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" | ||
| also CC the relevant Sig and Sig-Architecture | ||
|
|
||
|
|
||
| ### Test Metadata | ||
|
|
||
| Each conformance test must include the following piece of metadata | ||
| within its associated comment: | ||
|
|
||
| - `Release`: indicates the Kubernetes release that the test was added to the | ||
| conformance test suite. If the test was modified in subsequent releases | ||
| then those releases should be included as well (comma separated) | ||
| - `Testname`: a human readable short name of the test | ||
| - `Description`: a detailed description of the test. This field must describe | ||
| the required behaviour of the Kubernetes components being tested using | ||
| [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field | ||
| is meant to be a "specification" of the tested Kubernetes features, as | ||
| such, it must be detailed enough so that readers can fully understand | ||
| the aspects of Kubernetes that are being tested without having to read | ||
| the test's code directly. Additionally, this test should provide a clear | ||
| distinction between the parts of the test that are there for the purpose | ||
| of validating Kubernetes rather than simply infrastructure logic that | ||
| is necessary to setup, or clean up, the test. | ||
|
|
||
| ### Sample Conformance Test | ||
|
|
||
| The following snippet of code shows a sample conformance test's metadata: | ||
|
|
||
| ``` | ||
| /* | ||
| Release : v1.9 | ||
| Testname: Kubelet: log output | ||
| Description: By default the stdout and stderr from the process being | ||
| executed in a pod MUST be sent to the pod's logs. | ||
| */ | ||
| framework.ConformanceIt("it should print the output to logs", func() { | ||
| ... | ||
| }) | ||
| ``` | ||
|
|
||
| The corresponding portion of the Kubernetes Conformance Documentfor this test would then look | ||
| like this: | ||
|
|
||
| > | ||
| > ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) | ||
| > | ||
| > Release : v1.9 | ||
| > | ||
| > By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. | ||
|
|
||
| ### Reporting Conformance Test Results | ||
|
|
||
| Conformance test results, by provider and releases, can be viewed in the | ||
| federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conformance-all). | ||
| If you wish to contribute conformance test results for your provider, | ||
| please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#). | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does "1.3 development cluster" mean? A cluster built from the master branch after the 1.2 release was cut but before the 1.3 release was cut?
Note that kubectl officially only supports one release of skew.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
where are the docs on kubectl skew? I wasn't aware it was that limited
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not the most obvious location, I know, but the origin of this doc predates kubernetes.io:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah... I say the same thing whenever I have to refer to that uri.
These bullet points (written before docs were imported into k/community) were based on the "nodes" statement. Should we cut this back to just one release of skew?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will have a PR with this in it later today, trying to tighten up wording in a few other places as well