Skip to content

Commit 0a010b1

Browse files
YuviGoldYuviGold
authored
MGMT-7599: Add documentation on the repository CI (openshift#2584)
1 parent 8369c23 commit 0a010b1

File tree

4 files changed

+186
-95
lines changed

4 files changed

+186
-95
lines changed

‎CONTRIBUTING.md

+35-34
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,11 @@ Organized history and to create a CHANGE LOG for each version.
1616

1717
[This script](https://github.com/openshift/assisted-service/blob/master/hack/check-commit-message.sh#L7) checks for a valid issue (JIRA or GitHub) reference and fails the build otherwise with the message
1818

19-
```bash
20-
Your commit message is missing either a JIRA issue ('JIRA-1111'), a GitHub issue ('#39').
21-
You can also ignore the ticket checking with 'NO-ISSUE'.
19+
```text
20+
Your commit message should start with a JIRA issue ('JIRA-1111'), a GitHub issue ('#39'), or a BugZilla issue ('Bug 123')
21+
with a following colon(:).
22+
i.e. 'MGMT-42: Summary of the commit message'
23+
You can also ignore the ticket checking with 'NO-ISSUE' for master only.
2224
```
2325

2426
It will search for a commit message containing a valid JIRA/GitHub issue notation or "NO-ISSUE" in case there's no ticket.
@@ -31,51 +33,50 @@ The openshift-ci bot looks for `Bug XXX:` in the title of the pull request in or
3133

3234
1. JIRA reference without a Bugzilla link
3335

34-
```
35-
MGMT-6075 Implement ResetHostValidation API call
36-
```
36+
```text
37+
MGMT-6075: Implement ResetHostValidation API call
38+
```
3739
38-
2. No reference
40+
1. No reference
3941
40-
```
41-
NO-ISSUE allow GitHub-created reverts to be used
42-
```
42+
```text
43+
NO-ISSUE: allow GitHub-created reverts to be used
44+
```
4345
44-
3. Bugzilla and JIRA reference
46+
1. Bugzilla and JIRA reference
4547
46-
```
47-
Bug 1957227: Allow overriding defaults via provided ConfigMap
48+
```text
49+
Bug 1957227: Allow overriding defaults via provided ConfigMap
4850
49-
[...]
51+
[...]
5052
51-
Closes: OCPBUGSM-28781
52-
```
53+
Closes: OCPBUGSM-28781
54+
```
5355
54-
**NOTE**
56+
**NOTE**: For this commit the following is also correct, but the PR is not automatically linked with the bug tracker. Linking only to Bugzilla without referencing JIRA is not correct.
5557
56-
For this commit the following is also correct, but the PR is not automatically linked with the bug tracker. Linking only to Bugzilla without referencing JIRA is not correct.
58+
```text
59+
OCPBUGSM-28781: Allow overriding defaults via provided ConfigMap
60+
```
5761
58-
```
59-
OCPBUGSM-28781 Allow overriding defaults via provided ConfigMap
60-
```
62+
1. GitHub reference
6163
62-
4. GitHub reference
64+
```text
65+
#1 Fixing the very first GitHub issue
66+
```
6367
64-
```
65-
#1 Fixing the very first GitHub issue
66-
```
68+
**NOTE**: The following is also correct
6769
68-
**NOTE**
70+
```text
71+
Fixing the very first GitHub issue
6972
70-
The following is also correct
71-
72-
```
73-
Fixing the very first GitHub issue
73+
[...]
7474
75-
[...]
75+
Closes: #1
76+
```
7677
77-
Closes: #1
78-
```
78+
[PULL_REQUEST_TEMPLATE]: https://github.com/openshift/assisted-service/blob/master/docs/pull_request_template.md
7979
80+
## Testing
8081
81-
[PULL_REQUEST_TEMPLATE]: https://github.com/openshift/assisted-service/blob/master/docs/pull_request_template.md
82+
More information is available here: [Assisted Installer Testing](docs/dev/testing.md)

‎README.md

+2-61
Original file line numberDiff line numberDiff line change
@@ -72,68 +72,9 @@ After every change in the API (`swagger.yaml`) the code should be generated and
7272
skipper make generate-from-swagger
7373
```
7474

75-
## Test
75+
## Testing
7676

77-
The tests of assisted-service are divided into 3 categories:
78-
79-
* Unit tests - Focused on a module/function level while other modules are mocked.
80-
Unit tests are located next to a module named `module_name_test.go`
81-
* Subsystem tests - Focused on assisted-service component while mocking other component like agent/installer results.
82-
Subsystem tests requires deploying the assisted-service on a k8s cluster together with DB and storage services.
83-
The subsystem tests are located on the [subsystem](https://github.com/openshift/assisted-service/tree/master/subsystem) directory.
84-
* System tests (a.k.a e2e) - Running full flows with all components.
85-
The e2e tests are divided into u/s (upstream) basic workflows on [assisted-test-infra](https://github.com/openshift/assisted-test-infra/tree/master/discovery-infra/tests) and d/s (downstream) extended regression tests maintained by both DEV and QE teams on [kni-assisted-installer-auto](https://gitlab.cee.redhat.com/ocp-edge-qe/kni-assisted-installer-auto/-/tree/master/api_tests).
86-
87-
88-
### Subsystem tests pre-configuration
89-
90-
* Run [minikube](https://minikube.sigs.k8s.io/docs/start/) on your system
91-
* Enable [registry addon](https://minikube.sigs.k8s.io/docs/handbook/registry/) on your minikube:
92-
```bash
93-
minikube addons enable registry
94-
```
95-
* Set `LOCAL_ASSISTED_ORG` to point to your local registry address
96-
```bash
97-
export LOCAL_ASSISTED_ORG=localhost:5000
98-
```
99-
* Redirect the local registry to the minikube registry:
100-
```bash
101-
nohup kubectl port-forward svc/registry 5000:80 -n kube-system &>/dev/null &
102-
```
103-
* Make a tunnel to make minikube services reachable (the command will ask for root password):
104-
```bash
105-
nohup minikube tunnel &>/dev/null &
106-
```
107-
* Deploy services:
108-
```bash
109-
skipper make deploy-test
110-
```
111-
112-
### Run tests
113-
```shell
114-
skipper make [test|unit-test] [environment variables]
115-
```
116-
117-
* `test` - Runs subsystem tests.
118-
* `unit-test` - Runs unit tests.
119-
120-
Environment variables:
121-
122-
* `FOCUS="install_cluster"` - An optional flag used for [focused specs](https://onsi.github.io/ginkgo/#focused-specs) with regular expression.
123-
* `SKIP="install_cluster"` - An optional flag to skip scopes with regular expressions.
124-
* `TEST="./internal/host"` - An optional flag used for testing a specific package.
125-
* `VERBOSE=true` - An optional flag to print verbosed data.
126-
127-
128-
### Update service for the subsystem tests
129-
130-
if you are making changes and don't want to deploy everything once again you can simply run this command:
131-
132-
```shell
133-
skipper make patch-service
134-
```
135-
136-
It will build and push a new image of the service to your Docker registry, then delete the service pod from minikube, the deployment will handle the update and pull the new image to start the service again.
77+
More information is available here: [Assisted Installer Testing](docs/dev/testing.md)
13778

13879
## Update Discovery Image base OS
13980

‎docs/dev/running-test.md

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# How to run Assisted-service subsystem tests
2+
3+
Assisted-service subsystem tests require deploying the assisted-service on a k8s cluster together with DB and storage services.
4+
The subsystem tests are located on the [subsystem](https://github.com/openshift/assisted-service/tree/master/subsystem) directory.
5+
6+
## Subsystem tests pre-configuration
7+
8+
* Run [minikube](https://minikube.sigs.k8s.io/docs/start/) on your system
9+
* Enable [registry addon](https://minikube.sigs.k8s.io/docs/handbook/registry/) on your minikube:
10+
11+
```bash
12+
minikube start
13+
minikube addons enable registry
14+
```
15+
16+
* Set `LOCAL_ASSISTED_ORG` to point to your local registry address
17+
18+
```bash
19+
export LOCAL_ASSISTED_ORG=localhost:5000
20+
```
21+
22+
* Redirect the local registry to the minikube registry:
23+
24+
```bash
25+
nohup kubectl port-forward svc/registry 5000:80 -n kube-system &>/dev/null &
26+
```
27+
28+
* Make a tunnel to make minikube services reachable (the command will ask for root password):
29+
30+
```bash
31+
nohup minikube tunnel &>/dev/null &
32+
```
33+
34+
* Deploy services:
35+
36+
```bash
37+
skipper make deploy-test
38+
```
39+
40+
## Run tests
41+
42+
```bash
43+
skipper make [test|unit-test] [environment variables]
44+
```
45+
46+
* `test` - Runs subsystem tests.
47+
* `unit-test` - Runs unit tests.
48+
49+
Environment variables:
50+
51+
* `FOCUS="install_cluster"` - An optional flag used for [focused specs](https://onsi.github.io/ginkgo/#focused-specs) with regular expression.
52+
* `SKIP="install_cluster"` - An optional flag to skip scopes with regular expressions.
53+
* `TEST="./internal/host"` - An optional flag used for testing a specific package.
54+
* `VERBOSE=true` - An optional flag to print verbosed data.
55+
56+
## Update service for the subsystem tests
57+
58+
if you are making changes and don't want to deploy everything once again you can simply run this command:
59+
60+
```bash
61+
skipper make patch-service
62+
```
63+
64+
It will build and push a new image of the service to your Docker registry, then delete the service pod from minikube, the deployment will handle the update and pull the new image to start the service again.

‎docs/dev/testing.md

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Assisted Installer Testing
2+
3+
The assisted-installer tests are divided into 3 categories:
4+
5+
* **Unit tests** - Focused on a module/function level while other modules are mocked.
6+
Unit tests are located in the package, where the code they are testing resides, using the pattern `<module_name>_test.go`.
7+
8+
* **Subsystem tests** - Focused on the component while mocking other component.
9+
For example, assisted-service subsystem tests mock the agent responses.
10+
11+
* **System tests** (a.k.a e2e) - Running full flows with all components.
12+
The e2e tests are divided into u/s (upstream) basic workflows on [assisted-test-infra](https://github.com/openshift/assisted-test-infra/tree/master/discovery-infra/tests) and d/s (downstream) extended regression tests maintained by both DEV and QE teams on [kni-assisted-installer-auto](https://gitlab.cee.redhat.com/ocp-edge-qe/kni-assisted-installer-auto/-/tree/master/api_tests).
13+
14+
## Repository CI
15+
16+
Our CI jobs are currently managed and ran by two CI tools - a Jenkins hosted on <http://assisted-jenkins.usersys.redhat.com> and a Prow hosts on <https://prow.ci.openshift.org>.
17+
18+
| [Jenkins]((http://assisted-jenkins.usersys.redhat.com)) | [Prow](https://prow.ci.openshift.org) |
19+
|---|---|
20+
| Local for Assisted ecosystem | Company-wide |
21+
| Checks comments for JIRA | Runs e2e
22+
| Manages images in quay.io/ocpmetal | Runs all testing checks (lint, unit, etc)
23+
24+
Assisted-service CI jobs are defined under [openshift/release](https://github.com/openshift/release) repository on [openshift-assisted-service-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/assisted-service/openshift-assisted-service-master.yaml).
25+
Read more about OpenShift CI infrastructure on [OpenShift CI Docs](https://docs.ci.openshift.org/docs/).
26+
27+
All the currently available jobs for the openshift/assisted-service repository can be viewed on [Openshift CI Step Registry](https://steps.ci.openshift.org/search?job=openshift-assisted-service).
28+
29+
### Adding a new CI job
30+
31+
When adding a new job the following rules of thumbs should be taken into account:
32+
33+
* Test logic needs to be maintained in the repository under test and not under openshift/release.
34+
It would allow easier integration with other tools, less dependency of the CI infrastructure, and most importantly the availability to run it locally.
35+
36+
* When introducing a new job it should be both a presubmit job and a periodic job. A presubmit job needs to be available so contributors would be able to run it on their PRs before merging.
37+
38+
The presubmit job needs to be configured as `always_run: false` and `optional: true` (not blocking a merge) until proving stability.
39+
New OCP releases might break one of Assisted workflows since Assisted isn't part of OCP.
40+
41+
The periodic job needs to run on a frequent basis (e.g. daily) and have a `reporter_config` configured, in order to be notified on Slack whenever there's a breakage.
42+
43+
* In case the new job affects multiple repositories - every repository should have the same presubmit job so it could be tested for every component change.
44+
For example, you can see that the `e2e-metal-assisted-olm` job is defined on several different repositories in this [link](https://steps.ci.openshift.org/search?job=e2e-metal-assisted-olm).
45+
46+
[An example of a PR adding a new job](https://github.com/openshift/release/pull/21604)
47+
48+
### FAQ
49+
50+
#### **How can I debug CI failures?**
51+
52+
A CI job can be debugged only in runtime.
53+
Once a job terminates it can no longer be debugged because the cluster / machines used to run the job get torn down at the end of it.
54+
55+
However, each job produces artifacts such as (logs, SOS reports, must-gather logs) which can be used to try to analyze what went wrong in retrospect. Those artifacts can be accessed by going to the job artifacts.
56+
57+
You can follow the [OpenShift CI doc "Interact With Running Jobs"](https://docs.ci.openshift.org/docs/how-tos/interact-with-running-jobs/) guide or try to run the experimental [Debug Prow Jobs live](https://gist.github.com/omertuc/1ef4bdf22f0fedfbde46cf1feb149bb9) gist in order to connect to the OCP cluster running your prow tests. Contributions are welcome. :)
58+
59+
#### **If a CI job fails, where should I look for assisted-related failures?**
60+
61+
When a PR job fails there's a "details" button next to the GitHub context. It will show the [Spyglass](https://github.com/kubernetes/test-infra/tree/master/prow/spyglass) view. In there, you should look if there are other builds that failed recently for the same job using the "Job History" button.
62+
63+
#### **When is it ok to retest?**
64+
65+
Whenever there's a failure - first, you should look for its root cause before hitting the "/retest" command.
66+
It should only be used when there's a known flaky issue.
67+
Using the retest feature for no reason just wastes the project CI resources and money.
68+
69+
#### **How does the retest bot works?**
70+
71+
When a PR is ready to be merged (approved and not held) all the jobs will be retested for every new master that's being updated. In case any of the job fails - the openshift-bot will try to retest it automatically.
72+
The retest job is defined under [infra-periodics.yaml](https://github.com/openshift/release/blob/c121e55f68fb37af41d7cd16877eaa79eeb972f1/ci-operator/jobs/infra-periodics.yaml#L202-L241)
73+
74+
#### **Where do these jobs run?**
75+
76+
Depends on the job.
77+
78+
* Single-stage tests (e.g. lint, unit tests) run inside of a scheduled container. [Read more](https://docs.ci.openshift.org/docs/architecture/ci-operator/#declaring-tests)
79+
* Jobs that require a cluster (e.g. subsystem) run on a claimed OCP cluster from an hibernated pool of clusters.
80+
[Read more](https://docs.ci.openshift.org/docs/architecture/ci-operator/#testing-with-a-cluster-from-a-cluster-pool)
81+
* Baremetal jobs (i.e. e2e) run on a provisioned baremetal machine by [Equnix](https://www.equinix.nl/).
82+
83+
## How to run Assisted-service subsystem tests
84+
85+
More information is available here: [Assisted Installer Testing](docs/dev/testing.md).

0 commit comments

Comments
 (0)