-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Fix running integration tests as part of build #12693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,50 +1,36 @@ | ||
| import { exec } from "../../util/shell"; | ||
| import { Werft } from "../../util/werft"; | ||
| import { JobConfig } from "./job-config"; | ||
| import { PREVIEW_K3S_KUBECONFIG_PATH } from "./const"; | ||
|
|
||
| const phases = { | ||
| TRIGGER_INTEGRATION_TESTS: "trigger integration tests", | ||
| RUN_INTEGRATION_TESTS: "Run integration tests", | ||
| }; | ||
|
|
||
| /** | ||
| * Trigger integration tests | ||
| */ | ||
| export async function triggerIntegrationTests(werft: Werft, config: JobConfig, username: string) { | ||
| werft.phase(phases.TRIGGER_INTEGRATION_TESTS, "Trigger integration tests"); | ||
| export async function runIntegrationTests(werft: Werft, config: JobConfig, username: string) { | ||
| werft.phase(phases.RUN_INTEGRATION_TESTS, "Run integration tests"); | ||
|
|
||
| if (!config.withIntegrationTests) { | ||
| if (config.withIntegrationTests == "skip") { | ||
| // If we're skipping integration tests we wont trigger the job, which in turn won't create the | ||
| // ci/werft/run-integration-tests Github Check. As ci/werft/run-integration-tests is a required | ||
| // check this means you can't merge your PR without override checks. | ||
| werft.log(phases.TRIGGER_INTEGRATION_TESTS, "Skipped integration tests"); | ||
| werft.done(phases.TRIGGER_INTEGRATION_TESTS); | ||
| werft.log(phases.RUN_INTEGRATION_TESTS, "Skipped integration tests"); | ||
| werft.done(phases.RUN_INTEGRATION_TESTS); | ||
| return; | ||
| } | ||
|
|
||
| try { | ||
| const imageVersion = exec( | ||
| `docker run --rm eu.gcr.io/gitpod-core-dev/build/versions:${config.version} cat /versions.yaml | yq r - 'components.integrationTest.version'`, | ||
| { silent: true }, | ||
| ).stdout.trim(); | ||
|
|
||
| exec(`git config --global user.name "${username}"`); | ||
| const annotations = [ | ||
| `version="${imageVersion}"`, | ||
| `namespace="${config.previewEnvironment.namespace}"`, | ||
| `username="${username}"`, | ||
| `updateGitHubStatus="gitpod-io/gitpod"`, | ||
| ] | ||
| .map((annotation) => `-a ${annotation}`) | ||
| .join(" "); | ||
| exec(`werft run --remote-job-path .werft/run-integration-tests.yaml ${annotations} github`, { | ||
| slice: phases.TRIGGER_INTEGRATION_TESTS, | ||
| }).trim(); | ||
|
|
||
| werft.done(phases.TRIGGER_INTEGRATION_TESTS); | ||
| exec( | ||
| `KUBECONFIG="${PREVIEW_K3S_KUBECONFIG_PATH}" GOOGLE_APPLICATION_CREDENTIALS=/home/gitpod/.config/gcloud/legacy_credentials/[email protected]/adc.json /workspace/test/run.sh ${config.withIntegrationTests}`, | ||
| ); | ||
| werft.done(phases.RUN_INTEGRATION_TESTS); | ||
| } catch (err) { | ||
| if (!config.mainBuild) { | ||
| werft.fail(phases.TRIGGER_INTEGRATION_TESTS, err); | ||
| werft.fail(phases.RUN_INTEGRATION_TESTS, err); | ||
| } | ||
| exec("exit 0"); | ||
| throw err; | ||
| } | ||
| } |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,52 +21,64 @@ Such tests are for example: | |
|
|
||
| ### Automatically at Gitpod | ||
|
|
||
| There is a [werft job](../.werft/run-integration-tests.yaml) that runs the integration tests against `core-dev` preview environments. | ||
| You can opt-in to run the integrations tests as part of the build job. that runs the integration tests against preview environments. | ||
|
|
||
| > For tests that require an existing user the framework tries to automatically select one from the DB. | ||
| > - On preview envs make sure to create one before running tests against it! | ||
| > - If it's important to use a certain user (with fixed settings, for example) pass the additional `username` parameter. | ||
|
|
||
| Example command: | ||
| ``` | ||
| werft run github -j .werft/run-integration-tests.yaml -a namespace=staging-gpl-2658-int-tests -a version=gpl-2658-int-tests.57 -f | ||
|
|
||
| ```sh | ||
| werft job run github -a with-preview=true -a with-integration-tests=webapp -f | ||
| ``` | ||
|
|
||
| ### Manually | ||
|
|
||
| You may want to run tests to assert whether a Gitpod installation is successfully integrated. | ||
|
|
||
| #### Using a pod | ||
|
|
||
| Best for when you want to validate an environment. | ||
|
|
||
| 1. Update image name in `integration.yaml` for job `integration-job` to latest built by werft. | ||
| 2. Optionally add your username in that job argument or any other additional params. | ||
| 2. Apply yaml file that will add all necessary permissions and create a job that will run tests. | ||
| * [`kubectl apply -f ./integration.yaml`](./integration.yaml) | ||
| 3. Check logs to inspect test results like so `kubectl logs -f jobs/integration-job`. | ||
| 4. Tear down the integration user and job when testing is done. | ||
| * [`kubectl delete -f ./integration.yaml`](./integration.yaml) | ||
|
|
||
| #### Go test | ||
|
|
||
| This is best for when you're actively developing Gitpod. | ||
|
|
||
| Test will work if images that they use are already cached by Gitpod instance. If not, they might fail if it takes too long to pull an image. | ||
|
|
||
| There are 4 different types of tests: | ||
|
|
||
| 1. Enterprise specific, that require valid license to be installed. Run those with `-enterprise=true` | ||
| 2. Tests that require correct user (user should have github OAuth integration setup with gitpod). Run those with `-username=<gitpod_username>`. Make sure to load https://github.com/gitpod-io/gitpod-test-repo and https://github.com/gitpod-io/gitpod workspaces inside your gitpod that you are testing to preload those images onto your node. Wait for it to finish pulling those image, this will ensure that test will not fail due to timeout while waiting to pull an image for the first time. | ||
| 3. To test gitlab integration, add `-gitlab=true` | ||
| 4. All other tests. | ||
|
|
||
| To run the tests: | ||
| 1. Clone this repo (`git clone [email protected]:gitpod-io/gitpod.git`), and `cd` to `./gitpod/test` | ||
| 2. Run the tests like so | ||
| ```console | ||
| go test -v ./... \ | ||
| -kubeconfig=<path_to_kube_config_file> \ | ||
| -namespace=<namespace_where_gitpod_is_installed> \ | ||
| -username=<gitpod_user_with_oauth_setup> \ | ||
| -enterprise=<true|false> \ | ||
| -gitlab=<true|false> | ||
| ``` | ||
| 3. If you want to run specific test, add `-run <test>` before `-kubeconfig` parameter. | ||
| If you want to run an entire test suite, the easiest is to use `./test/run.sh`: | ||
|
|
||
| ```sh | ||
| # This will run all test suites | ||
| ./test/run.sh | ||
|
|
||
| # This will run only the webapp test suite | ||
| ./test/run.sh webapp | ||
| ``` | ||
|
|
||
| If you're iterating on a single test, the easiest is to use `go test` directly. If your integration tests depends on having having a user token available, then you'll have to set USER_TOKEN manually (see run.sh on how to fetch the credentials that are used during our build) | ||
|
|
||
| ```sh | ||
| cd test | ||
| go test -v ./... \ | ||
| -run <test> \ | ||
| -kubeconfig=/home/gitpod/.kube/config \ | ||
| -namespace=default \ | ||
| -username=<gitpod_user_with_oauth_setup> \ | ||
| -enterprise=<true|false> \ | ||
| -gitlab=<true|false> | ||
| ``` | ||
|
|
||
| A concrete example would be | ||
|
|
||
| ```sh | ||
| cd test | ||
| go test -v ./... \ | ||
| -run TestAdminBlockUser \ | ||
| -kubeconfig=/home/gitpod/.kube/config \ | ||
| -namespace=default | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we default to
skiphere, but for empty-string/unknown we default toall?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's mainly about how we want to interpret how people use (or not use) the
with-integration-testsjob attribute, especially when used in GH comments/PR descriptions where you can use/werft run with-integration-testsor if you just set the attribute in your PR description (/werft with-integration-tests- in both cases without specifying a value - which case the value is the empty string.In the CLI it's less common but you could do
werft job run github -a with-integration-tests="".So if it is null/undefined it means they didn't use the
with-integration-testsattribute at all - if it's the empty string they did use it, but just didn't specify a value - I decided to interpret that as "they want to run all the tests"There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be useful to add this explanation as a comment in the code, or do you think it is overkill?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a log line of
As I think that will
s)