chore(ci): prevent commit including generated files #379

eunjae-lee · 2022-04-14T16:07:59Z

🧭 What and Why

🎟 JIRA Ticket: https://algolia.atlassian.net/browse/APIC-427

Changes included:

This PR includes pre-commit hook to check if any generated file is staged. If so, it throws and the commit is canceled. The hook doesn't automatically unstage the file, but informs which files are caught.

There is some tricky case to cover, so had to write either JS/TS instead of bash script.

And I chose JS here, to run it as quickly as possible. ts-node is too slow to be run every time we run git commit.

TODO

check the commit message, and if it contains [skip ci], then it should really skip this test too. Otherwise, commits like this will fail.

🧪 Test

CI

netlify · 2022-04-14T16:08:04Z

✅ Deploy Preview for api-clients-automation canceled.

Name	Link
🔨 Latest commit	`32dd38e`
🔍 Latest deploy log	https://app.netlify.com/sites/api-clients-automation/deploys/6262ae12c865fd00093e12d3

scripts/ci/husky/pre-commit.js

scripts/ci/husky/__tests__/pre-commit.test.js

algolia-bot · 2022-04-14T16:18:30Z

✗ The generated branch has been deleted.

If the PR has been merged, you can check the generated code on the main branch.

.husky/pre-commit

millotp

This is perfect thank you ! it will solve all of our process issues 🙏

package.json

scripts/ci/husky/pre-commit.js

Co-authored-by: Pierre Millot <[email protected]>

millotp · 2022-04-19T08:57:46Z

It can be hard when we have a lot of files to remove the correct files, like when we rename a folder or something, can you add a little script that uses the same logic just to unstage generated files pls ?

shortcuts

love it!! GG

scripts/ci/husky/pre-commit.js

scripts/ci/husky/__tests__/pre-commit.test.js

scripts/ci/husky/pre-commit.js

shortcuts · 2022-04-20T14:44:46Z

Maybe now that we have a defined list of files generated and not generated we can improve the delete models function and delete all the generated code before generation

(I've made an update on the codegen removing on #386)

…tion into chore/githook

millotp

Such a nice improvement !

millotp · 2022-04-22T08:30:49Z

scripts/ci/husky/pre-commit.js

+  '!clients/**/.openapi-generator-ignore',
+
+  // Java
+  '!clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/exceptions/*',


Can you update those path pls ? They now use algoliasearch-core/src/com/algolia...

millotp · 2022-04-22T08:32:59Z

scripts/ci/husky/pre-commit.js

+  // Java
+  '!clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/exceptions/*',
+  '!clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/utils/*',
+  'clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/utils/echo/EchoResponse.java',


The echo response is a bit more complicated now, its:

Suggested change

'clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/utils/echo/EchoResponse.java',

'clients/algoliasearch-client-java-2/algoliasearch-core/src/com/algolia/utils/echo/EchoResponse*.java',

'!clients/algoliasearch-client-java-2/algoliasearch-core/src/com/algolia/utils/echo/EchoResponseInterface.java',

millotp · 2022-04-22T08:34:02Z

scripts/ci/husky/pre-commit.js

+const execa = require('execa');
+const micromatch = require('micromatch');
+
+const GENERATED_FILE_PATTERNS = [


This list is meant to be updated whenever we create a file by hand, maybe it would be nice to have it easily accessible in the config folder, wdyt ?

(and maybe documented somewhere on how to update this list)

I created config/generation.config.js. What do you think?

I created a json first, but that way we cannot add any comment. This config particularly will require us to write some contexts as comment. So I made it a js file. Let me know what you think. 73528a0

I think it's perfect like that !

@millotp great! and I applied the changes to the patterns in that js file.

millotp · 2022-04-22T08:36:08Z

scripts/ci/husky/pre-commit.js

+  const negativeMatchers = [];
+
+  patterns.forEach((pattern) => {
+    if (pattern.startsWith('!')) {


I'm not sure this simplification of splitting negative and positive will work for every case, for example if we have a sequence of [positive, negative, positive, negative], it will loose the pattern, but I don't have a better solution 🤷

Yes, I'm not too sure either. But I couldn't think of any edge case so far. Once we face it in the future, it will reveal itself and still won't ruin anything. Then we can adjust the logic to cover the case.

millotp · 2022-04-22T08:36:43Z

scripts/ci/husky/pre-commit.js

+      continue;
+    }
+
+    console.log(


really nice output 😃

shortcuts

Really nice! Good to go after Pierre's suggestions

shortcuts · 2022-04-22T08:39:50Z

scripts/ci/husky/__tests__/pre-commit.test.js

@@ -0,0 +1,24 @@
+/* eslint-disable import/no-commonjs */


Test can still be in TS 😇

shortcuts · 2022-04-22T08:40:04Z

scripts/ci/husky/pre-commit.js

+const execa = require('execa');
+const micromatch = require('micromatch');
+
+const GENERATED_FILE_PATTERNS = [


shortcuts · 2022-04-22T13:26:46Z

@eunjae-lee can you bump the cache key? idk why it works here but not on main

shortcuts

That's nice!!!! GG

shortcuts · 2022-04-22T14:01:46Z

config/generation.config.js

@@ -0,0 +1,30 @@
+// eslint-disable-next-line import/no-commonjs
+module.exports = {


We can maybe add a description of what this file does exactly

shortcuts

Actually the doc is enough for the file so feel free to ignore

* chore(ci): add pre-commit hook to unstage generated codes * chore: test commit * chore: implement pre-commit script * chore: fix lint error * chore: fix lint error * chore: call js file directly * chore: update .cache_version * Update scripts/ci/husky/pre-commit.js Co-authored-by: Pierre Millot <[email protected]> * Update scripts/ci/husky/pre-commit.js Co-authored-by: Pierre Millot <[email protected]> * chore: use postinstall instead of prepare * chore: better error message * chore: fix patterns * chore: unstage files automatically and exit with 0 * chore: change loop * chore: skip deleted files * chore: split config to a separate file * chore: update docs * chore: update cache version Co-authored-by: Pierre Millot <[email protected]>

eunjae-lee added 3 commits April 12, 2022 15:31

chore(ci): add pre-commit hook to unstage generated codes

3e75735

chore: test commit

227edcc

chore: implement pre-commit script

9167fc9

chore: fix lint error

b9d1057

eunjae-lee commented Apr 14, 2022

View reviewed changes

scripts/ci/husky/pre-commit.js Show resolved Hide resolved

eunjae-lee commented Apr 14, 2022

View reviewed changes

scripts/ci/husky/__tests__/pre-commit.test.js Show resolved Hide resolved

Merge branch 'main' into chore/githook

ff0b1b8

chore: fix lint error

054c031

millotp reviewed Apr 15, 2022

View reviewed changes

.husky/pre-commit Outdated Show resolved Hide resolved

eunjae-lee added 3 commits April 15, 2022 10:33

chore: call js file directly

3fe4723

chore: update .cache_version

40cbd5e

Merge branch 'main' into chore/githook

c5d7781

eunjae-lee marked this pull request as ready for review April 15, 2022 12:12

eunjae-lee requested review from a team, damcou and shortcuts and removed request for a team April 15, 2022 12:12

millotp reviewed Apr 15, 2022

View reviewed changes

eunjae-lee and others added 5 commits April 19, 2022 10:24

Merge branch 'main' into chore/githook

fbfba2b

Update scripts/ci/husky/pre-commit.js

c8c3c6e

Co-authored-by: Pierre Millot <[email protected]>

Update scripts/ci/husky/pre-commit.js

0b07721

Co-authored-by: Pierre Millot <[email protected]>

chore: use postinstall instead of prepare

4dc5a01

chore: better error message

202169c

shortcuts reviewed Apr 19, 2022

View reviewed changes

scripts/ci/husky/pre-commit.js Outdated Show resolved Hide resolved

eunjae-lee added 2 commits April 19, 2022 15:39

chore: fix patterns

6d34e74

chore: unstage files automatically and exit with 0

62c75fe

eunjae-lee added 4 commits April 21, 2022 16:29

chore: change loop

fc73866

Merge branch 'main' into chore/githook

958ff64

chore: skip deleted files

d5fa011

Merge branch 'chore/githook' of github.com:algolia/api-clients-automa…

958c1fa

…tion into chore/githook

millotp reviewed Apr 22, 2022

View reviewed changes

shortcuts reviewed Apr 22, 2022

View reviewed changes

eunjae-lee added 4 commits April 22, 2022 14:13

Merge branch 'main' into chore/githook

613fca9

chore: split config to a separate file

73528a0

chore: update docs

b7e752e

Merge branch 'main' into chore/githook

bb700fd

chore: update cache version

32dd38e

eunjae-lee requested a review from shortcuts April 22, 2022 14:00

shortcuts reviewed Apr 22, 2022

View reviewed changes

shortcuts approved these changes Apr 22, 2022

View reviewed changes

eunjae-lee merged commit 9fbba0e into main Apr 22, 2022

eunjae-lee deleted the chore/githook branch April 22, 2022 14:23

This was referenced Apr 28, 2022

chore: release 2022-04-28 #445

Closed

chore: release 2022-04-28 #446

Closed

chore: release 2022-04-29 #456

Closed

chore: release 2022-04-29 #461

Closed

	'clients/algoliasearch-client-java-2/algoliasearch-core/com/algolia/utils/echo/EchoResponse.java',
	'clients/algoliasearch-client-java-2/algoliasearch-core/src/com/algolia/utils/echo/EchoResponse*.java',
	'!clients/algoliasearch-client-java-2/algoliasearch-core/src/com/algolia/utils/echo/EchoResponseInterface.java',

		@@ -0,0 +1,30 @@
		// eslint-disable-next-line import/no-commonjs
		module.exports = {

chore(ci): prevent commit including generated files #379

chore(ci): prevent commit including generated files #379

Uh oh!

Conversation

eunjae-lee commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧭 What and Why

Changes included:

TODO

🧪 Test

Uh oh!

netlify bot commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for api-clients-automation canceled.

Uh oh!

Uh oh!

Uh oh!

algolia-bot commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✗ The generated branch has been deleted.

Uh oh!

Uh oh!

millotp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

millotp commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shortcuts left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shortcuts commented Apr 20, 2022

Uh oh!

millotp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

millotp Apr 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eunjae-lee Apr 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shortcuts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eunjae-lee commented Apr 14, 2022 •

edited

Loading

netlify bot commented Apr 14, 2022 •

edited

Loading

algolia-bot commented Apr 14, 2022 •

edited

Loading

millotp commented Apr 19, 2022 •

edited

Loading

millotp Apr 22, 2022 •

edited

Loading

eunjae-lee Apr 22, 2022 •

edited

Loading