ci: add npm run conformance #374

grant · 2021-10-26T23:54:23Z

Adds command to run conformance tests:

npm run conformance

Very similar to the Go setup, with a few minor modifications.

Conformance package-lock.json is ignored, because it's just pointing to file:../../.

FULL LOGS

npm run conformance

> @google-cloud/[email protected] conformance
> ./run_conformance_tests.sh

Defaulting to latest client.
Use './run_conformance_tests vX.X.X' to specify a specific release version.

up to date, audited 3 packages in 866ms

found 0 vulnerabilities

========== INSTALLING CLIENT@latest ==========
Note: only works with Go 1.16+ by default, see run_conformance_tests.sh for more information.
Done installing client@latest

========== HTTP CONFORMANCE TESTS ==========
2021/10/26 16:45:24 Validating for http...
2021/10/26 16:45:24 Framework server started.
2021/10/26 16:45:29 HTTP validation started...
2021/10/26 16:45:29 HTTP validation passed!
2021/10/26 16:45:29 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt.
2021/10/26 16:45:29 All validation passed!

========== BACKGROUND EVENT CONFORMANCE TESTS ==========
2021/10/26 16:45:29 Validating for legacyevent...
2021/10/26 16:45:29 Framework server started.
2021/10/26 16:45:34 Legacy event validation with legacy event requests...
2021/10/26 16:45:34 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:34 Legacy event validation with CloudEvent requests...
2021/10/26 16:45:34 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:34 Legacy event validation passed!
2021/10/26 16:45:34 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt.
2021/10/26 16:45:34 All validation passed!

========== CLOUDEVENT CONFORMANCE TESTS ==========
2021/10/26 16:45:34 Validating for cloudevent...
2021/10/26 16:45:34 Framework server started.
2021/10/26 16:45:39 CloudEvent validation with CloudEvent requests...
2021/10/26 16:45:39 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:39 CloudEvent validation with legacy event requests...
2021/10/26 16:45:39 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:39 CloudEvent validation passed!
2021/10/26 16:45:39 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt.
2021/10/26 16:45:39 All validation passed!

========== DECLARATIVE HTTP CONFORMANCE TESTS ==========
2021/10/26 16:45:39 Validating for http...
2021/10/26 16:45:39 Framework server started.
2021/10/26 16:45:44 HTTP validation started...
2021/10/26 16:45:44 HTTP validation passed!
2021/10/26 16:45:44 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt.
2021/10/26 16:45:44 All validation passed!

========== DECLARATIVE CLOUDEVENT CONFORMANCE TESTS ==========
2021/10/26 16:45:44 Validating for cloudevent...
2021/10/26 16:45:44 Framework server started.
2021/10/26 16:45:49 CloudEvent validation with CloudEvent requests...
2021/10/26 16:45:49 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:49 CloudEvent validation with legacy event requests...
2021/10/26 16:45:49 Events tried:
        - firebase-auth (PASSED)
        - firebase-db1 (PASSED)
        - firebase-db2 (PASSED)
        - firebase-db3 (PASSED)
        - firebase-db4 (PASSED)
        - firebase-db5 (PASSED)
        - firebase-db6 (PASSED)
        - firebase-db7 (PASSED)
        - firebase-db8 (PASSED)
        - firebase-dbdelete1 (PASSED)
        - firebase-dbdelete2 (PASSED)
        - firestore_complex (PASSED)
        - firestore_simple (PASSED)
        - legacy_pubsub (PASSED)
        - pubsub_binary (PASSED)
        - pubsub_text (PASSED)
        - storage (PASSED)
2021/10/26 16:45:49 CloudEvent validation passed!
2021/10/26 16:45:49 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt.
2021/10/26 16:45:49 All validation passed!

Signed-off-by: Grant Timmerman <[email protected]>

anniefu · 2021-10-26T23:57:22Z

run_conformance_tests.sh

+set -e
+
+# Navigate to the conformance tests and install the local FF.
+cd test/conformance;


[nit] May want to put the rest of this in a subshell by adding parenthesis around the whole thing. If you run this from repo root, this would put you in this directory after, right?

When running this from root (or even any directory within the npm package), I don't see the directory changing.

I believe npm scripts spawn a shell process under the hood, so npm run X will start a new shell, run, then exit.

Signed-off-by: Grant Timmerman <[email protected]>

matthewrobertson · 2021-10-27T00:09:07Z

CONTRIBUTING.md

@@ -34,6 +34,14 @@ To run an individual test, you can run a command such as the following:
 npm run test -- -g 'loading function'
 ```

+### Conformance Tests
+
+To run the conformance tests, run:


Should we add a note about needing to install Go 1.16 first?

Sounds good. I added a note there.

Signed-off-by: Grant Timmerman <[email protected]>

matthewrobertson

LGTM

grant added 2 commits October 26, 2021 16:46

ci: add npm run conformance

c5f58d7

Signed-off-by: Grant Timmerman <[email protected]>

ci: ignore lockfile for conformance tests

4847d07

Signed-off-by: Grant Timmerman <[email protected]>

grant self-assigned this Oct 26, 2021

google-cla bot added the cla: yes label Oct 26, 2021

grant requested a review from anniefu October 26, 2021 23:54

anniefu approved these changes Oct 26, 2021

View reviewed changes

docs: update contributing with conformance tests note

ab9b64b

Signed-off-by: Grant Timmerman <[email protected]>

matthewrobertson reviewed Oct 27, 2021

View reviewed changes

docs: add note about installing go for conformance tests

7c45fee

Signed-off-by: Grant Timmerman <[email protected]>

grant requested a review from matthewrobertson October 27, 2021 00:16

matthewrobertson approved these changes Oct 27, 2021

View reviewed changes

Merge branch 'master' into grant_conformance_local

28a8b0e

grant added the automerge Summon MOG for automerging label Oct 27, 2021

gcf-merge-on-green bot merged commit 8110d3a into master Oct 27, 2021

gcf-merge-on-green bot removed the automerge Summon MOG for automerging label Oct 27, 2021

grant deleted the grant_conformance_local branch November 18, 2021 23:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add npm run conformance #374

ci: add npm run conformance #374

grant commented Oct 26, 2021

anniefu Oct 26, 2021

grant Oct 27, 2021

matthewrobertson Oct 27, 2021

grant Oct 27, 2021

matthewrobertson left a comment

ci: add npm run conformance #374

ci: add npm run conformance #374

Conversation

grant commented Oct 26, 2021

anniefu Oct 26, 2021

Choose a reason for hiding this comment

grant Oct 27, 2021

Choose a reason for hiding this comment

matthewrobertson Oct 27, 2021

Choose a reason for hiding this comment

grant Oct 27, 2021

Choose a reason for hiding this comment

matthewrobertson left a comment

Choose a reason for hiding this comment