Ensure the $ARTIFACT_DIR variable is respected #2721
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: timflannagan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
| @@ -116,8 +117,7 @@ func (ctx TestContext) DumpNamespaceArtifacts(namespace string) error { | |||
| "KUBECONFIG=" + kubeconfigPath, | |||
| } | |||
|
|
|||
| // compiled test binary running e2e tests is run from the root ./bin directory | |||
| cmd := exec.Command("../test/e2e/collect-ci-artifacts.sh") | |||
There was a problem hiding this comment.
I ran into a sourcing issue when running the make e2e command locally as it's working directory points to ./test/e2e, so there's some skew between the working directory when using the kind or kubeconfig provisioners.
test/e2e/e2e_test.go
Outdated
| if collectArtifactsScriptPath != nil && *collectArtifactsScriptPath != "" { | ||
| os.Setenv("E2E_ARTIFACTS_SCRIPT", *collectArtifactsScriptPath) | ||
| } |
There was a problem hiding this comment.
Note: there wasn't any clear way of injecting global state to the ctx package, and there weren't any setter methods so I wanted to avoid introducing that pattern, so we're using an internal environment variable to basically act as a cross-package variable. This is admittedly a bit hacky of an implementation so I'm open to suggestions.
| @@ -137,6 +137,17 @@ func setDerivedFields(ctx *TestContext) error { | |||
| return fmt.Errorf("nil RESTClient") | |||
| } | |||
|
|
|||
| if ctx.artifactsDir == "" { | |||
There was a problem hiding this comment.
Could this poorly interact with some downstream only code: https://github.com/openshift/operator-framework-olm/blob/master/staging/operator-lifecycle-manager/test/e2e/e2e_test.go#L69?
There was a problem hiding this comment.
We should be in the clear there as we're building up the junit reporter for ginkgo vs. telling either of the provisioner implementation where to dump testing artifacts when configured.
There was a problem hiding this comment.
Also: we could bubble up that downstream-only change to make upstream backwards-compatible to eliminate any skew between implementations (which might be relevant for any effort around ginkgo v1 -> v2)?
|
I'll try and come back to this at some point. /hold |
openshift-ci
bot
added
the
do-not-merge/hold
timflannagan
force-pushed
the
ci/respect-artifacts-dir
branch
from
06b234a to
ad497aa
Compare
|
Ran these changes locally after rebasing against master: $ make e2e ARTIFACT_DIR=./tmp E2E_INSTALL_NS=olm TEST="Not found APIs"
...
$ tree test/e2e/tmp
test/e2e/tmp
├── deprecated-e2e-pvt9s
│ ├── get_catalogsources_-o_yaml
│ ├── get_clusterserviceversions_-o_yaml
│ ├── get_events_--sort-by_.lastTimestamp
│ ├── get_installplans_-o_yaml
│ ├── get_operatorgroups_-o_yaml
│ ├── get_pods_-o_wide
│ └── get_subscriptions_-o_yaml
└── junit
└── junit_e2e_01.xmlAnd I had updated one of the test cases in the deprecated e2e suite that's being focused on in that example to purposely fail: - Expect(ctx.Ctx().Client().Status().Update(context.Background(), ip)).To(Succeed(), "failed to update the resource")
+ Expect(ctx.Ctx().Client().Status().Update(context.Background(), ip)).ToNot(Succeed(), "failed to update the resource") |
| if collectArtifactsScriptPath != nil && *collectArtifactsScriptPath != "" { | ||
| os.Setenv("E2E_ARTIFACT_SCRIPT", *collectArtifactsScriptPath) | ||
| } | ||
|
|
There was a problem hiding this comment.
Note: just a hack to inject state so the ctx package can derive where to find the artifacts bash script path location.
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
|
Can we correct the title, commit message and all references to |
timflannagan
changed the title
This directory is populating when the e2e suite is supplied with an output $ARTIFACT_DIR, and the script that's responsible for gathering CI testing artifacts when a test case fails. Signed-off-by: timflannagan <[email protected]>
Signed-off-by: timflannagan <[email protected]>
timflannagan
force-pushed
the
ci/respect-artifacts-dir
branch
from
ad497aa to
f82e274
Compare
Description of the change:
#2432 had initially added support for CI to gather testing artifacts when the $ARTIFACT_DIR (#2729 renamed this variable from $ARTIFACTS_DIR from the original implementation) was specified, and the kind provisioner was responsible for running a bash script that fired off a series of kubectl commands at the namespace level. This functionality works fine for CI environments that are configured to use the
make e2e-localtarget, but this kind of functionality wasn't emulated to also work with the kubeconfig provisioner that's used under-the-hood when users runmake e2edirectly.These changes aim to consolidate both of those provisioner implementations so that both of them gathering testing artifacts when that variable has been specified, and an individual test case has failed.
Motivation for the change:
Ensure both the kubeconfig and kind provisioner(s) are able to collecting testing artifacts.
Reviewer Checklist
/doc[FLAKE]are truly flaky[FLAKE]tag are no longer flaky