Introduce a mechanism for gathering test artifacts during individual test failures #2442
Conversation
| } | ||
|
|
||
| // compiled test binary running e2e tests is run from the root ./bin directory | ||
| cmd := exec.Command("../test/e2e/collect-ci-artifacts.sh") |
There was a problem hiding this comment.
Note: likely need to stat this file vs. relying on a local file reference.
| if currentTest.Failed { | ||
| log("collecting the %s namespace artifacts as the '%s' test case failed", ns.GetName(), currentTest.TestText) | ||
| if err := ctx.Ctx().DumpNamespaceArtifacts(ns.GetName()); err != nil { | ||
| log("failed to collect namespace artifacts: %v", err) |
There was a problem hiding this comment.
We avoid performing operations like Expect(err).ToBe(Nil()) here as we want the namespace to be deleted in any case, regardless of whether this method fails to gather artifacts.
| echo "Storing the test artifact output in the ${TEST_ARTIFACTS_DIR} directory" | ||
| for command in "${commands[@]}"; do | ||
| echo "Collecting ${command} output..." | ||
| COMMAND_OUTPUT_FILE=${TEST_ARTIFACTS_DIR}/${command// /_} |
There was a problem hiding this comment.
This output file likely needs some work - here's an example of an e2e run locally:
$ make e2e-local ARTIFACTS_DIR=/tmp/artifacts
...
$ tree /tmp/artifacts/catalog-e2e-2jq84/
/tmp/artifacts/catalog-e2e-2jq84/
├── get_clusterserviceversions_-o_yaml
├── get_events_--sort-by_.lastTimestamp
├── get_installplans_-o_yaml
├── get_operatorgroups_-o_yaml
├── get_pods_-o_wide
└── get_subscriptions_-o_yaml| } | ||
| ctx.Logf("collecting logs in the %s artifacts directory", ctx.artifactsDir) | ||
|
|
||
| logDir := filepath.Join(ctx.artifactsDir, namespace) |
There was a problem hiding this comment.
I'm open to suggestions on how to best dump individual test failure artifacts. In the current implementation, this creates a directory named after whatever namespace was generated for that test. It wouldn't be immediately clear how to map this to an individual test failure, outside of looking at the overall CI logs that are produced during a run (as the namespace generated is logged).
| for command in "${commands[@]}"; do | ||
| echo "Collecting ${command} output..." | ||
| COMMAND_OUTPUT_FILE=${TEST_ARTIFACTS_DIR}/${command// /_} | ||
| kubectl -n ${TEST_NAMESPACE} ${command} >> "${COMMAND_OUTPUT_FILE}" |
There was a problem hiding this comment.
Note: this script would also neglect deleting any "empty" files here - any file that contains an empty List will still be created and housed in this directory:
apiVersion: v1
items: []
kind: List
metadata:
resourceVersion: ""
selfLink: ""
…st failures Update the testing e2e suite and add a mechanism for gathering test artifacts during individual test failures. Currently, container logs are gathered when deprovisioning upstream kind clusters, yet we lack fine-grain ability to diagnose test failures further. Note: test failures use the `CurrentGinkgoTestDescription.Failed` field to determine failures. Testing artifacts are only gathered when the base $ARTIFACTS_DIR environment variable has been specified. Add a collect-ci-artifacts.sh bash script, responsible for gathering OLM native resources for an individual testing namespace. This bash script will be called when tearing down the generated testing namespace for relevant e2e packages. Currently, the artifact gathering process is restricted to only a single namespace - longer term, it might be possible to instead migrate towards collecting resources that all share a similar label selector, and utilizing that label selector to handle multi-namespace testing scenarios. Introduce another helper function in test/e2e/util_test.go that's responsible for gathering test artifacts (i.e. calling this newly introduced script) when the test case had failed, and in either case, delete the namespace. Signed-off-by: timflannagan <[email protected]>
…mespace Signed-off-by: timflannagan <[email protected]>
timflannagan
force-pushed
the
test/collect-failed-test-artifacts
branch
from
29d4377 to
95ba037
Compare
|
/approve |
openshift-ci
bot
added
the
approved
timflannagan
changed the title
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: anik120, kevinrizza, timflannagan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Update the testing e2e suite and add a mechanism for gathering test
artifacts during individual test failures. Currently, container logs are
gathered when deprovisioning upstream kind clusters, yet we lack
fine-grain ability to diagnose test failures further.
Note: test failures use the
CurrentGinkgoTestDescription.Failedfieldto determine failures. Testing artifacts are only gathered when the base
$ARTIFACTS_DIR environment variable has been specified.
Add a collect-ci-artifacts.sh bash script, responsible for gathering OLM
native resources for an individual testing namespace. This bash script
will be called when tearing down the generated testing namespace for
relevant e2e packages. Currently, the artifact gathering process is
restricted to only a single namespace - longer term, it might be
possible to instead migrate towards collecting resources that all share
a similar label selector, and utilizing that label selector to handle
multi-namespace testing scenarios.
Introduce another helper function in test/e2e/util_test.go that's
responsible for gathering test artifacts (i.e. calling this newly
introduced script) when the test case had failed, and in either case,
delete the namespace.
Signed-off-by: timflannagan [email protected]
Reviewer Checklist
/doc