Skip to content

feat(script): convert matrix scripts #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 4, 2022
Merged

feat(script): convert matrix scripts #318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
045b8a1
feat(script): convert matrix scripts
shortcuts Mar 31, 2022
5aeae9b
maybe less duplication
shortcuts Mar 31, 2022
658cbe0
Merge branch 'main' into feat/convert-matrix-scripts
shortcuts Mar 31, 2022
f6e794d
apply changes from suggestion
shortcuts Apr 4, 2022
7f4728c
remove scripts from cache
shortcuts Apr 4, 2022
965cdbd
oops
shortcuts Apr 4, 2022
4545b78
Merge branch 'main' into feat/convert-matrix-scripts
shortcuts Apr 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 120 additions & 17 deletions .github/actions/cache/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,7 @@ runs:
'clients/algoliasearch-client-javascript/packages/algoliasearch/**',
'clients/algoliasearch-client-javascript/packages/client-search/**',
'clients/algoliasearch-client-javascript/packages/client-analytics/**',
'clients/algoliasearch-client-javascript/packages/client-personalization/**',
'templates/javascript/**'
'clients/algoliasearch-client-javascript/packages/client-personalization/**'
)}}

- name: Restore built JavaScript search client
Expand All @@ -207,7 +206,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-search/builds/**',
'clients/algoliasearch-client-javascript/packages/client-search/package.json',
'specs/bundled/search.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript recommend client
Expand All @@ -223,7 +223,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/recommend/builds/**',
'clients/algoliasearch-client-javascript/packages/recommend/package.json',
'specs/bundled/recommend.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript query-suggestions client
Expand All @@ -239,7 +240,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-query-suggestions/builds/**',
'clients/algoliasearch-client-javascript/packages/client-query-suggestions/package.json',
'specs/bundled/query-suggestions.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript personalization client
Expand All @@ -255,7 +257,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-personalization/builds/**',
'clients/algoliasearch-client-javascript/packages/client-personalization/package.json',
'specs/bundled/personalization.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript analytics client
Expand All @@ -271,7 +274,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-analytics/builds/**',
'clients/algoliasearch-client-javascript/packages/client-analytics/package.json',
'specs/bundled/analytics.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript abtesting client
Expand All @@ -287,7 +291,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-abtesting/builds/**',
'clients/algoliasearch-client-javascript/packages/client-abtesting/package.json',
'specs/bundled/abtesting.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript insights client
Expand All @@ -303,7 +308,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-insights/builds/**',
'clients/algoliasearch-client-javascript/packages/client-insights/package.json',
'specs/bundled/insights.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript sources client
Expand All @@ -319,7 +325,8 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-sources/builds/**',
'clients/algoliasearch-client-javascript/packages/client-sources/package.json',
'specs/bundled/sources.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Restore built JavaScript predict client
Expand All @@ -335,34 +342,130 @@ runs:
'clients/algoliasearch-client-javascript/packages/client-predict/builds/**',
'clients/algoliasearch-client-javascript/packages/client-predict/package.json',
'specs/bundled/predict.yml',
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

# Restore Java clients: used during 'cts' or 'codegen'
- name: Restore built Java client
- name: Restore built Java search client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-java-2
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-java-2/**',
'clients/algoliasearch-client-java-2/search/**',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also include generators and scripts, but maybe we need a better system, you can see the dependencies here

Copy link
Member Author

@shortcuts shortcuts Apr 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will need to at least re-arrange our folder structure for the CTS and Scripts in order to be able to scope those more easily, I'll put the cache to the scripts folder for now but this should be in a generation/build related folder

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is node_modules in scripts, mostly because of different versions in our package.json. I'll fix it in an other PR and add back the keys 😓

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but changes to package version should also invalidate the cache

'clients/algoliasearch-client-java-2/model/search/**',
'specs/bundled/search.yml',
'templates/java/**',
'specs/bundled/search.yml'
'generators/src/**'
)}}

# Restore PHP clients: used during 'cts' or 'codegen'
- name: Restore built PHP client
- name: Restore built PHP search client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/SearchApi.php',
'clients/algoliasearch-client-php/lib/Configuration/SearchConfig.php',
'specs/bundled/search.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP recommend client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/RecommendApi.php',
'clients/algoliasearch-client-php/lib/Configuration/RecommendConfig.php',
'specs/bundled/recommend.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP personalization client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/PersonalizationApi.php',
'clients/algoliasearch-client-php/lib/Configuration/PersonalizationConfig.php',
'specs/bundled/personalization.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP analytics client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/*',
'specs/bundled/search.yml'
'clients/algoliasearch-client-php/lib/Api/AnalyticsApi.php',
'clients/algoliasearch-client-php/lib/Configuration/AnalyticsConfig.php',
'specs/bundled/analytics.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP insights client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/InsightsApi.php',
'clients/algoliasearch-client-php/lib/Configuration/InsightsConfig.php',
'specs/bundled/insights.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP abtesting client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/AbTestingApi.php',
'clients/algoliasearch-client-php/lib/Configuration/AbTestingConfig.php',
'specs/bundled/abtesting.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Restore built PHP query-suggestions client
if: ${{ inputs.job == 'cts' || inputs.job == 'codegen' }}
uses: actions/cache@v2
with:
path: clients/algoliasearch-client-php
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
'clients/algoliasearch-client-php/lib/Api/QuerySuggestionsApi.php',
'clients/algoliasearch-client-php/lib/Configuration/QuerySuggestionsConfig.php',
'specs/bundled/query-suggestions.yml',
'templates/php/**',
'generators/src/**'
)}}

- name: Install JavaScript dependencies
Expand Down
8 changes: 4 additions & 4 deletions .github/actions/setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ runs:
run: |
base_changed=${{ steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 }}

matrix=$(./scripts/ci/create-spec-matrix.sh $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }})
matrix=$(yarn workspace scripts createMatrix $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }})

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
Expand All @@ -101,7 +101,7 @@ runs:
algoliasearch_changed=${{ steps.diff.outputs.JS_ALGOLIASEARCH_CHANGED > 0 }}
common_changed=${{ steps.diff.outputs.JS_COMMON_CHANGED > 0 }}

matrix=$(./scripts/ci/create-client-matrix.sh javascript $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }})
matrix=$(yarn workspace scripts createMatrix $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }} javascript)

if [[ $algoliasearch_changed == 'true' || $base_changed == 'true' ]]; then
echo "Running algoliasearch: true"
Expand Down Expand Up @@ -132,7 +132,7 @@ runs:
run: |
base_changed=${{ steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.GENERATORS_CHANGED > 0 || steps.diff.outputs.JAVA_TEMPLATE_CHANGED > 0 }}

matrix=$(./scripts/ci/create-client-matrix.sh java $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }})
matrix=$(yarn workspace scripts createMatrix $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }} java)

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
Expand All @@ -153,7 +153,7 @@ runs:
run: |
base_changed=${{ steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.GENERATORS_CHANGED > 0 || steps.diff.outputs.PHP_TEMPLATE_CHANGED > 0 }}

matrix=$(./scripts/ci/create-client-matrix.sh php $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }})
matrix=$(yarn workspace scripts createMatrix $base_changed ${{ steps.diff.outputs.ORIGIN_BRANCH }} php)

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
Expand Down
19 changes: 12 additions & 7 deletions .github/workflows/check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,8 @@ jobs:
format('{0}/builds/**', matrix.client.folder),
format('{0}/package.json', matrix.client.folder),
format('specs/bundled/{0}.yml', matrix.client.name),
'templates/javascript/**'
'templates/javascript/**',
'generators/src/**'
)}}

- name: Generate '${{ matrix.client.name }}' client
Expand Down Expand Up @@ -210,8 +211,7 @@ jobs:
'clients/algoliasearch-client-javascript/packages/algoliasearch/**',
'clients/algoliasearch-client-javascript/packages/client-search/**',
'clients/algoliasearch-client-javascript/packages/client-analytics/**',
'clients/algoliasearch-client-javascript/packages/client-personalization/**',
'templates/javascript/**'
'clients/algoliasearch-client-javascript/packages/client-personalization/**'
)}}

- name: Build 'algoliasearch' client
Expand Down Expand Up @@ -248,9 +248,11 @@ jobs:
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
format('{0}/**', matrix.client.folder),
format('{0}/{1}/**', matrix.client.folder, matrix.client.name),
format('{0}/model/{1}/**', matrix.client.folder, matrix.client.name),
format('specs/bundled/{0}.yml', matrix.client.name),
'templates/java/**',
format('specs/bundled/{0}.yml', matrix.client.name)
'generators/src/**'
)}}

- name: Generate '${{ matrix.client.name }}' client
Expand Down Expand Up @@ -291,8 +293,11 @@ jobs:
key: |
${{ env.CACHE_VERSION }}-${{
hashFiles(
format('{0}/**', matrix.client.folder),
format('specs/bundled/{0}.yml', matrix.client.name)
format('{0}/lib/Api/{1}.php', matrix.client.folder, matrix.client.api),
format('{0}/lib/Configuration/{1}.php', matrix.client.folder, matrix.client.config),
format('specs/bundled/{0}.yml', matrix.client.name),
'templates/php/**',
'generators/src/**'
)}}

- name: Generate '${{ matrix.client.name }}' client
Expand Down
41 changes: 0 additions & 41 deletions scripts/ci/create-client-matrix.sh
View file
Open in desktop

This file was deleted.

29 changes: 0 additions & 29 deletions scripts/ci/create-spec-matrix.sh
View file
Open in desktop

This file was deleted.

Loading