Skip to content

Caching process update #2467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Dec 12, 2021
Merged

Caching process update #2467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2c0212b
CI: workflows: disable stack where it is not used
Anton-Latukha Dec 11, 2021
d0cd643
CI: {bench,hackage} add Linux platform config
Anton-Latukha Dec 11, 2021
ee64445
CI: {caching,test,bench,hackage}: retrieving index timestamp
Anton-Latukha Dec 11, 2021
a4fce1f
CI: {caching,test,bench,hackage}: new caching process
Anton-Latukha Dec 11, 2021
999c872
CI: caching: separate source retrieval step & separate build step
Anton-Latukha Dec 11, 2021
6483d96
CI: {caching,bench}: add executable patching for cache
Anton-Latukha Dec 11, 2021
b27b36d
Merge branch 'master' into 2021-12-11-ci-upd-caching
mergify[bot] Dec 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 39 additions & 8 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,22 +47,53 @@ jobs:
- run: git fetch origin master # check the master branch for benchmarking

- uses: haskell/actions/setup@v1
id: HaskEnvSetup
with:
ghc-version : ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}
enable-stack: false

- name: Cache Cabal
- name: Linux Platform config
run: |
echo "CABAL_PKGS_DIR=~/.cabal/packages" >> $GITHUB_ENV

# All workflows which distinquishes cache on `cabal.project` needs this.
- name: Workaround shorten binary names
run: |
sed -i.bak -e 's/haskell-language-server/hls/g' \
-e 's/haskell_language_server/hls/g' \
haskell-language-server.cabal cabal.project
sed -i.bak -e 's/Paths_haskell_language_server/Paths_hls/g' \
src/**/*.hs exe/*.hs

- name: Retrieving `cabal.project` Hackage timestamp
run: |
# Form: index-state: 2021-11-29T08:11:08Z
INDEX_STATE_ENTRY=$(grep index-state cabal.project)
# Form: 2021-11-29T08-11-08Z
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV

- name: Hackage sources cache
uses: actions/cache@v2
env:
cache-name: hackage-sources
with:
path: |
~/.cabal/packages
~/.cabal/store
key: v2-${{ runner.os }}-${{ matrix.ghc }}-bench-${{ hashFiles('cabal.project') }}
path: ${{ env.CABAL_PKGS_DIR }}
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}
restore-keys: ${{ env.cache-name }}-

- name: Compiled deps cache
uses: actions/cache@v2
env:
cache-name: compiled-deps
with:
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
restore-keys: |
v2-${{ runner.os }}-${{ matrix.ghc }}-build-${{ hashFiles('cabal.project') }}
v2-${{ runner.os }}-${{ matrix.ghc }}-bench-
v2-${{ runner.os }}-${{ matrix.ghc }}
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
${{ env.cache-name }}-${{ runner.os }}-

- run: cabal update

Expand Down
1 change: 1 addition & 0 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
with:
ghc-version : ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}
enable-stack: false

# some alpines come with integer-simple instead of integer-gmp
- name: Force integer-simple
Expand Down
73 changes: 59 additions & 14 deletions .github/workflows/caching.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ on:
branches:
- master

env:
cabalBuild: "v2-build all --enable-tests --enable-benchmarks"

jobs:

pre_job:
Expand Down Expand Up @@ -66,19 +69,19 @@ jobs:
- uses: actions/checkout@v2

- uses: haskell/actions/setup@v1
id: HaskEnvSetup
with:
ghc-version: ${{ matrix.ghc }}
ghc-version : ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}
enable-stack: false

- if: runner.os == 'Windows'
name: (Windows) Platform config
run: |
echo "CABAL_STORE_DIR=$SYSTEMDRIVE\\SR" >> $GITHUB_ENV
echo "CABAL_PKGS_DIR=~\\AppData\\cabal\\packages" >> $GITHUB_ENV
- if: ( runner.os == 'Linux' ) || ( runner.os == 'macOS' )
name: (Linux,macOS) Platform config
run: |
echo "CABAL_STORE_DIR=~/.cabal/store" >> $GITHUB_ENV
echo "CABAL_PKGS_DIR=~/.cabal/packages" >> $GITHUB_ENV

# Needs to be before Cache Cabal so the cache can detect changes to the modified cabal.project file
Expand All @@ -92,22 +95,64 @@ jobs:
echo "package floskell" >> cabal.project
echo " ghc-options: -O0" >> cabal.project

- name: Cache Cabal
# Shorten binary names as a workaround for filepath length limits in Windows,
# but since tests are hardcoded on this workaround -
# all platforms (in 2021-12-07) need it.
# All workflows which distinquishes cache on `cabal.project` needs this.
- name: Workaround shorten binary names
run: |
sed -i.bak -e 's/haskell-language-server/hls/g' \
-e 's/haskell_language_server/hls/g' \
haskell-language-server.cabal cabal.project
sed -i.bak -e 's/Paths_haskell_language_server/Paths_hls/g' \
src/**/*.hs exe/*.hs

- name: Retrieving `cabal.project` Hackage timestamp
run: |
# Form: index-state: 2021-11-29T08:11:08Z
INDEX_STATE_ENTRY=$(grep index-state cabal.project)
# Form: 2021-11-29T08-11-08Z
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV

# 2021-12-02: NOTE: Cabal Hackage source tree storage does not depend on OS or GHC really,
# but can depend on `base`.
# But this caching is happens only inside `master` for `master` purposes of compiling the deps
# so having a shared pool here that depends only on Hackage pin & does not depend on `base` is "good enough"
# & used such because it preserves 10% of a global cache storage pool.
- name: Hackage sources cache
uses: actions/cache@v2
env:
cache-name: hackage-sources
with:
path: ${{ env.CABAL_PKGS_DIR }}
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}
restore-keys: ${{ env.cache-name }}-

- name: Compiled deps cache
uses: actions/cache@v2
env:
cache-name: cache-cabal
cache-name: compiled-deps
with:
path: |
${{ env.CABAL_PKGS_DIR }}
${{ env.CABAL_STORE_DIR }}
key: v2-${{ runner.os }}-${{ matrix.ghc }}-build-${{ hashFiles('cabal.project') }}
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
restore-keys: |
v2-${{ runner.os }}-${{ matrix.ghc }}-bench-${{ hashFiles('cabal.project') }}
v2-${{ runner.os }}-${{ matrix.ghc }}-build-
v2-${{ runner.os }}-${{ matrix.ghc }}
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
${{ env.cache-name }}-${{ runner.os }}-

- run: cabal update

- name: Download all sources
run: |
cabal $cabalBuild --only-download

# repeating builds to workaround segfaults in windows and ghc-8.8.4
- name: Build
run: cabal build --only-dependencies || cabal build --only-dependencies || cabal build --only-dependencies
# This build agenda in not to have successful code,
# but to cache what can be cached, so step is fault tolerant & would always succseed.
# 2021-12-11: NOTE: Building all targets, since
# current Cabal does not allow `all --enable-tests --enable-benchmarks --only-dependencies`
- name: Build all targets; try 3 times
continue-on-error: true
run: |
cabal $cabalBuild || cabal $cabalBuild || cabal $cabalBuild
39 changes: 30 additions & 9 deletions .github/workflows/hackage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,23 +49,44 @@ jobs:
- uses: actions/checkout@v2

- uses: haskell/actions/setup@v1
id: HaskEnvSetup
with:
ghc-version : ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}
enable-stack: false

- name: Cache Cabal
- name: Linux Platform config
run: |
echo "CABAL_PKGS_DIR=~/.cabal/packages" >> $GITHUB_ENV

- name: Retrieving `cabal.project` Hackage timestamp
run: |
# Form: index-state: 2021-11-29T08:11:08Z
INDEX_STATE_ENTRY=$(grep index-state cabal.project)
# Form: 2021-11-29T08-11-08Z
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV

- name: Hackage sources cache
uses: actions/cache@v2
env:
cache-name: hackage-sources
with:
path: ${{ env.CABAL_PKGS_DIR }}
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}
restore-keys: ${{ env.cache-name }}-

- name: Compiled deps cache
uses: actions/cache@v2
env:
cache-name: cache-cabal
cache-name: compiled-deps
with:
path: |
~/.cabal/packages
~/.cabal/store
key: v2-${{ runner.os }}-${{ matrix.ghc }}-build-${{ hashFiles('cabal.project') }}
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
restore-keys: |
v2-${{ runner.os }}-${{ matrix.ghc }}-bench-${{ hashFiles('cabal.project') }}
v2-${{ runner.os }}-${{ matrix.ghc }}-build-
v2-${{ runner.os }}-${{ matrix.ghc }}
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
${{ env.cache-name }}-${{ runner.os }}-

- name: "Run cabal check"
run: |
Expand Down
41 changes: 27 additions & 14 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,19 +76,19 @@ jobs:
- uses: actions/checkout@v2

- uses: haskell/actions/setup@v1
id: HaskEnvSetup
with:
ghc-version : ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}
enable-stack: false

- if: runner.os == 'Windows'
name: (Windows) Platform config
run: |
echo "CABAL_STORE_DIR=$SYSTEMDRIVE\\SR" >> $GITHUB_ENV
echo "CABAL_PKGS_DIR=~\\AppData\\cabal\\packages" >> $GITHUB_ENV
- if: ( runner.os == 'Linux' ) || ( runner.os == 'macOS' )
name: (Linux,macOS) Platform config
run: |
echo "CABAL_STORE_DIR=~/.cabal/store" >> $GITHUB_ENV
echo "CABAL_PKGS_DIR=~/.cabal/packages" >> $GITHUB_ENV

# Needs to be before Cache Cabal so the cache can detect changes to the modified cabal.project file
Expand All @@ -102,9 +102,7 @@ jobs:
echo "package floskell" >> cabal.project
echo " ghc-options: -O0" >> cabal.project

# Shorten binary names as a workaround for filepath length limits in Windows,
# but since tests are hardcoded on this workaround -
# all platforms (in 2021-12-07) need it.
# All workflows which distinquishes cache on `cabal.project` needs this.
- name: Workaround shorten binary names
run: |
sed -i.bak -e 's/haskell-language-server/hls/g' \
Expand All @@ -113,19 +111,34 @@ jobs:
sed -i.bak -e 's/Paths_haskell_language_server/Paths_hls/g' \
src/**/*.hs exe/*.hs

- name: Cache Cabal
- name: Retrieving `cabal.project` Hackage timestamp
run: |
# Form: index-state: 2021-11-29T08:11:08Z
INDEX_STATE_ENTRY=$(grep index-state cabal.project)
# Form: 2021-11-29T08-11-08Z
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV

- name: Hackage sources cache
uses: actions/cache@v2
env:
cache-name: hackage-sources
with:
path: ${{ env.CABAL_PKGS_DIR }}
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}
restore-keys: ${{ env.cache-name }}-

- name: Compiled deps cache
uses: actions/cache@v2
env:
cache-name: cache-cabal
cache-name: compiled-deps
with:
path: |
${{ env.CABAL_PKGS_DIR }}
${{ env.CABAL_STORE_DIR }}
key: v2-${{ runner.os }}-${{ matrix.ghc }}-build-${{ hashFiles('cabal.project') }}
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
restore-keys: |
v2-${{ runner.os }}-${{ matrix.ghc }}-bench-${{ hashFiles('cabal.project') }}
v2-${{ runner.os }}-${{ matrix.ghc }}-build-
v2-${{ runner.os }}-${{ matrix.ghc }}
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
${{ env.cache-name }}-${{ runner.os }}-

- run: cabal update

Expand Down