Skip to content

docs: overhaul for scripts and their end-to-end tests #1187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 4, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions .github/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,5 +210,37 @@ The `pnpm run test:migrate` script is run in CI to ensure that templating change
See `.github/workflows/test-migrate.yml`.

> Tip: if the migration test is failing in CI and you don't see any errors, try [downloading the full logs](https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/using-workflow-run-logs#downloading-logs).
> There'll likely be a list of changed files under a message like _`Oh no! Running the migrate script modified some files:`_.
> You can also try running the test script locally.

##### Migration Snapshot Failures

The migration test uses the [Vitest file snapshot](https://vitest.dev/guide/snapshot#file-snapshots) in `script/__snapshots__/migrate-test-e2e.js.snap` to store expected differences to this repository after running the migration script.
The end-to-end migration test will fail any changes that don't keep the same differences in that snapshot.

You can update the snapshot file by:

1. Committing any changes to your local repository
2. Running `pnpm i` and `pnpm build` if any updates have been made to the `package.json` or `src/` files, respectively
3. Running `pnpm run test:migrate -u` to update the snapshot

At this point there will be some files changed:

- `script/__snapshots__/migrate-test-e2e.js.snap` will have updates if any files mismatched templates
- The actual updated files on disk will be there too

If the snapshot file changes are what you expected, then you can commit them.
The rest of the file changes can be reverted.

> [🚀 Feature: Add a way to apply known file changes after migration #1184](https://github.com/JoshuaKGoldberg/create-typescript-app/issues/1184) tracks turning the test snapshot into a feature.

##### Unexpected File Modifications

The migration test also asserts that no files were unexpectedly changed.
If you see a failure like:

```plaintext
Oh no! Running the migrate script unexpectedly modified:
- ...
```

...then that means the file generated from templates differs from what's checked into the repository.
This is most often caused by changes to templates not being applied to checked-in files too.
3 changes: 3 additions & 0 deletions script/create-test-e2e.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ const title = "Test Title";

await $`rm -rf ${repository}`;

// Fist we run with --mode create to create a new new local repository,
// asserting that pnpm i passes in that repository's directory.
await $({
stdio: "inherit",
})`c8 -o ./coverage-create -r html -r lcov --src src node ./bin/index.js --base everything --mode create --author ${author} --email ${email} --description ${description} --owner ${owner} --title ${title} --repository ${repository} --skip-all-contributors-api --skip-github-api`;
Expand All @@ -18,6 +20,7 @@ process.chdir(repository);

const failures = [];

// Then we run each of the CI commands to assert that they pass too.
for (const command of [
`pnpm i`,
`pnpm run build`,
Expand Down
14 changes: 9 additions & 5 deletions script/initialize-test-e2e.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ const owner = "RNR1";
const title = "New Title Test";
const repository = "new-repository-test";

// First we run initialize to modifies the local repo, so we can test the changes
// Fist we run with --mode initialize to modify the local repository files,
// asserting that the created package.json keeps the general description.
await $({
stdio: "inherit",
})`node ./bin/index.js --description ${description} --base everything --mode initialize --owner ${owner} --title ${title} --repository ${repository} --skip-all-contributors-api --skip-github-api --skip-restore`;
})`pnpm run initialize --base everything --mode initialize --description ${description} --owner ${owner} --title ${title} --repository ${repository} --skip-all-contributors-api --skip-github-api --skip-restore`;

const newPackageJson = JSON.parse(
(await fs.readFile("./package.json")).toString(),
Expand All @@ -21,6 +22,8 @@ console.log("New package JSON:", newPackageJson);
assert.equal(newPackageJson.description, description);
assert.equal(newPackageJson.name, repository);

// Assert that the initialize script used the provided values in files,
// except for the 'This package was templated with ...' attribution notice.
const files = await globby(["*.*", "**/*.*"], {
gitignore: true,
ignoreFiles: ["script/initialize-test-e2e.js"],
Expand All @@ -34,11 +37,12 @@ for (const search of [`/JoshuaKGoldberg/`, "create-typescript-app"]) {
);
}

// Use Knip to assert that none of the template-only dependencies remain.
// They should have been removed as part of initialization.
try {
await $`pnpm run lint:knip`;
} catch (error) {
console.error("Error running lint:knip:", error);
process.exitCode = 1;
throw new Error("Error running lint:knip:", { cause: error });
}

// Now that initialize has passed normal steps, we reset everything,
Expand All @@ -49,4 +53,4 @@ await $`pnpm i`;
await $`pnpm run build`;
await $({
stdio: "inherit",
})`c8 -o ./coverage -r html -r lcov --src src node ./bin/index.js --base everything --description ${description} --mode initialize --owner ${owner} --title ${title} --repository ${repository} --skip-all-contributors-api --skip-github-api --skip-removal --skip-restore`;
})`c8 -o ./coverage -r html -r lcov --src src node ./bin/index.js --base everything --mode initialize --description ${description}--owner ${owner} --title ${title} --repository ${repository} --skip-all-contributors-api --skip-github-api --skip-removal --skip-restore`;
20 changes: 9 additions & 11 deletions script/migrate-test-e2e.js
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@ describe("expected file changes", () => {
`Looks like there were no changes to ${file} from migration?`,
);

// If this fails, see .github/DEVELOPMENT.md > Setup Scripts for context.
// Then see .github/DEVELOPMENT.md > Migration Snapshot Failures.
expect(contentsAfterGitMarkers).toMatchSnapshot();
});
});
Expand Down Expand Up @@ -149,22 +151,18 @@ test("unexpected file changes", async () => {
const gitDiffCommand = `git diff HEAD -- ${unstagedModifiedFiles.join(
" ",
)}`;
console.log(
`Stdout from running \`${gitDiffCommand}\`:\n${
(await execaCommand(gitDiffCommand)).stdout
}`,
);
const { stdout } = await execaCommand(gitDiffCommand);

console.log(`Stdout from running \`${gitDiffCommand}\`:\n${stdout}`);

throw new Error(
[
"",
"Oh no! Running the migrate script modified some files:",
"Oh no! Running the migrate script unexpectedly modified:",
...unstagedModifiedFiles.map((filePath) => ` - ${filePath}`),
"",
"That likely indicates changes made to the repository without",
"corresponding updates to templates in src/.",
"",
"Please search for those file(s)' name(s) under src/migrate for",
"the corresponding template and update those as well.",
"See .github/DEVELOPMENT.md > Setup Scripts for context.",
"Then see .github/DEVELOPMENT.md > Unexpected File Modifications.",
]
.map((line) => chalk.red(line))
.join("\n"),
Expand Down