Skip to content

Commit fe903de

Browse files
davidcavazosdavidcavazos
authored
docs: document setup files (#3996)
* document setup files * add comments to update readme * add link * document hard timeout * add example directory structure * add tip for concurrent runs * expand unique resources section
1 parent 31f4616 commit fe903de

File tree

3 files changed

+224
-7
lines changed

3 files changed

+224
-7
lines changed

‎.github/config/README.md

+211
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# Node.js setup
2+
3+
The CI setup file is an _optional_ file to configure the testing infrastructure runtime environment on a per-package basis.
4+
It _must_ be named `ci-setup.json` and be located in the same directory as the `package.json` file.
5+
6+
For example:
7+
8+
```sh
9+
my-product/
10+
└─ my-package/
11+
├─ package.json # package file
12+
└─ ci-setup.json # setup file
13+
```
14+
15+
## Default values
16+
17+
Here are the default values to be used if a `ci-setup.json` file is not present, or only some fields are defined.
18+
19+
```js
20+
{
21+
"env": {},
22+
"secrets": {},
23+
"node-version": 20,
24+
"timeout-minutes": 10,
25+
}
26+
```
27+
28+
These are defined in the [`nodejs-prod.jsonc`](nodejs-prod.jsonc) config file.
29+
30+
### Environment variables
31+
32+
Environment variables are defined as a dictionary where the key is the environment variable name, and the value is the variable’s value.
33+
34+
For example:
35+
36+
```js
37+
{
38+
"env": {
39+
"MY_VAR": "my-value",
40+
"MY_OTHER_VAR": "my-other-value"
41+
}
42+
}
43+
```
44+
45+
The test infrastructure then exports them as environment variables.
46+
47+
### Automatic environment variables
48+
49+
Additionally to the environment variables you define in the `ci-setup.json` file, the test infrastructure always exports the following environment variables:
50+
51+
- `PROJECT_ID`: The project used by the testing infrastructure.
52+
- `RUN_ID`: A random unique identifier, different on every test run.
53+
54+
> **Note**: An environment variable explicitly defined under `env` with the same name as an automatic variable overrides the automatic variable.
55+
56+
### Environment variable interpolation
57+
58+
Environment variables support using variable interpolation in the format `$VAR` or `${VAR}`.
59+
Secrets do not support this to avoid accidentally exposing secrets.
60+
61+
This can be useful for fully qualified resource names like `projects/$PROJECT_ID/resources/my-resource-id`, or to create unique resource names like `my-resource-$RUN_ID`.
62+
63+
## Unique resources
64+
65+
If a test creates and destroys their own resources, make sure the resource names are different for each run.
66+
This can be done by appending a unique identifier to each resource ID. We can use the `$RUN_ID` environment variable.
67+
68+
For example:
69+
70+
```js
71+
{
72+
"env": {
73+
"DATABASE": "projects/$PROJECT_ID/databases/my-db-$RUN_ID"
74+
}
75+
}
76+
```
77+
78+
For more information, see
79+
[Concurrent runs](https://github.com/GoogleCloudPlatform/cloud-samples-tools/blob/main/docs/testing-guidelines.md#concurrent-runs)
80+
in our testing guidelines.
81+
82+
### Secrets
83+
84+
Secrets can be defined as a dictionary where the key is the environment variable to which it’s exported to, and the value is the
85+
[Secret Manager](https://cloud.google.com/security/products/secret-manager)
86+
secret ID in the format `project-id/secret-id`.
87+
88+
For example:
89+
90+
```js
91+
{
92+
"secrets": {
93+
"MY_SECRET": "$PROJECT_ID/my-secret-id",
94+
"MY_OTHER_SECRET": "$PROJECT_ID/my-other-secret-id"
95+
}
96+
}
97+
```
98+
99+
The test infrastructure then fetches the actual secret data and exports them as environment variables.
100+
101+
### Secrets vs environment variables
102+
103+
Most configuration should be directly set with environment variables.
104+
This includes some things which might appear to be sensitive, but aren't.
105+
Minimizing secrets helps reduce maintenance and troubleshooting efforts.
106+
107+
For example, when a secret gets exposed, we must go through all the secrets used on that repository, assess which ones are real secrets and rotate them, and provide a justificatin of secrets that weren't rotated because they're just configurations.
108+
Keeping configurations as environment variables reduces the number of secrets needed to be assessed.
109+
110+
Examples of environment variables:
111+
112+
- Project IDs
113+
- Cloud resource IDs (or ID templates for a set of tests)
114+
- Service accounts emails (they look like email addresses, but aren't)
115+
- Usernames and passwords for _private_ resources (meaning anyone without project permissions cannot access those resources, so they can’t even reach the login)
116+
117+
These items are also likely to appear in logs, and that is OK.
118+
119+
Some things really are secret and if exposed, would create opportunities for unauthorized access, fraud, etc.
120+
The test infrastructure uses [Google Cloud Secret Manager](https://cloud.google.com/security/products/secret-manager) to store and retrieve secrets.
121+
122+
Examples of real secrets:
123+
124+
- API keys
125+
- Auth tokens
126+
- Identity tokens
127+
- Service account credentials (the JSON content with the actual keys)
128+
- Passwords or API keys to 3rd party services needed for testing
129+
- Usernames and passwords for _public_ resources (accessible to anyone)
130+
131+
> **Important**: Test code and scripts should _**never log secret values**_.
132+
> A test run printing a secret will stay in the commit history, even if another commit is made after.
133+
> This requires rotating that secret.
134+
135+
> **Note**: These testing environment variables and secrets are completely separate from [GitHub Actions secrets](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) and variables.
136+
> GitHub Actions secrets and variables are _not_ used by the testing infrastructure.
137+
138+
### Node version
139+
140+
You can opt in to use a different Node version if you need a specific version functionality.
141+
142+
For example:
143+
144+
```js
145+
{
146+
"_justification": "My reason for using a specific Node version.",
147+
"node-version": 18,
148+
}
149+
```
150+
151+
We **strongly recommend** not overriding this unless it's a version-specific feature or test.
152+
153+
This _might_ raise flags during review.
154+
155+
If you have a valid reason to use a different Node version, please provide a justification as a `_comment`.
156+
157+
> **Note**: In the future, we might implement automatic checks to validate that tests are using the recommended default version.
158+
159+
### Test timeout
160+
161+
If your tests take longer than the default test timeout, you can opt in to override it.
162+
163+
For example:
164+
165+
```js
166+
{
167+
"_justification": "My reason for this test to take longer to run.",
168+
"timeout-minutes": 30,
169+
}
170+
```
171+
172+
> **Note**: Overriding this **will** raise flags during review.
173+
174+
Tests should run fast, and the default timeout should be more than enough for most tests to run.
175+
If your tests are taking longer, consider splitting them into subpackages.
176+
177+
For example:
178+
179+
```sh
180+
my-product/
181+
├─ snippets/
182+
│ ├─ package.json
183+
│ ├─ fast-samples.js
184+
│ ├─ test/
185+
│ │ └─ fast-samples.test.js
186+
│ └─ slow-snippet/
187+
│ ├─ package.json
188+
│ ├─ slow-snippet.js
189+
│ └─ test/
190+
│ └─ slow-snippet.test.js
191+
├─ new-feature/
192+
│ ├─ package.json
193+
│ ├─ new-feature.js
194+
│ └─ test/
195+
│ └─ new-feature.test.js
196+
└─ e2e-sample/
197+
├─ package.json
198+
├─ slow-sample.js
199+
└─ test/
200+
└─ slow-sample.test.js
201+
```
202+
203+
For more information, see
204+
[Keep tests fast](https://github.com/GoogleCloudPlatform/cloud-samples-tools/blob/main/docs/testing-guidelines.md#keep-tests-fast)
205+
in our testing guidelines.
206+
207+
If you have a valid reason to use a different timeout, please provide a justification as a `_comment`.
208+
209+
There is a hard limit of 120 minutes that cannot be overriden.
210+
211+
> **Note**: In the future, we might implement automatic checks to validate that tests run fast to safely reduce the default timeout.

‎.github/config/nodejs-dev.jsonc

+7-4
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,15 @@
1717
{
1818
"package-file": [ "package.json" ],
1919
"ci-setup-filename": "ci-setup.json",
20+
21+
// If these change, please update the .github/config/README.md too!
2022
"ci-setup-defaults": {
21-
"node-version": 20,
22-
"timeout-minutes": 10,
2323
"env": { },
24-
"secrets": { }
24+
"secrets": { },
25+
"node-version": 20,
26+
"timeout-minutes": 10
2527
},
28+
2629
"ignore": [
2730
".github/config/", // prevent changes to exclusions from running all tests
2831
".eslintignore",
@@ -42,7 +45,7 @@
4245
".kokoro/",
4346
".prettierignore",
4447
".prettierrc.js",
45-
"cloud-samples-tools", // checked out by GH action in ci-*.yml
48+
"cloud-samples-tools", // checked out by GH action in ci-*.yml
4649
"CODEOWNERS",
4750
"CODE_OF_CONDUCT.md",
4851
"CONTRIBUTING.md",

‎.github/config/nodejs-prod.jsonc

+6-3
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,15 @@
1717
{
1818
"package-file": [ "package.json" ],
1919
"ci-setup-filename": "ci-setup.json",
20+
21+
// If these change, please update the .github/config/README.md too!
2022
"ci-setup-defaults": {
21-
"node-version": 20,
22-
"timeout-minutes": 10,
2323
"env": { },
24-
"secrets": { }
24+
"secrets": { },
25+
"node-version": 20,
26+
"timeout-minutes": 10
2527
},
28+
2629
"ignore": [
2730
".github/config/", // prevent changes to exclusions from running all tests
2831
".eslintignore",

0 commit comments

Comments
 (0)