-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Fix gh actions #1128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix gh actions #1128
Conversation
* fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * remove 4gotten trigger * Update-gh-actions (#6) * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * use macos-latest * try to update conda before creating mac env * better conda update trial * re-pin streamlit version * re-added trigger to run workflow in current branch * try to find out if conda mac env could be updated * install cmake, protobuf and rust b4 conda * add yes to conda update * lets try anaconda3-2022.05 * try environment.yml for mac as well * reenable conda mac env, add pip install also fix gitignore by changing from dream to invoke * remove - unecesary virtualenv creation - conda update change != macos back to == linux * remove cmake from brew install since pre-installed * disable opencv-python pip requirement * fixed commands to find latest package versions * update requirements for mac env * back to the roots - only install conda env depending on runner_os with or without extra env variables * check out macOS in azure-pipelines since becoming kind of tired of the GitHub Runner which is broken as ... * let's try to setup python and update conda env * initialize conda before using it * add trigger in azure-pipelines.yml * And another go for update first .... * update azure-pipelines.yml - add caching - add checkpoint download - add paths to trigger and more * unquote checkpoint-url * fix chekpoint-url variable * mkdir before downloading model * set pr trigger to main, rename anaconda cache * unique cacheHitVariables * try to use macos-latest instead of macos-12 * update test-invoke-conda.yml: - remove unecesarry echo step - use s-weigand/setup-conda@v1 - remove conda update from install deps step since updated with action * update test-invoke-conda.yml: - rename conda env cache from ldm to invokeai - reorder steps: 1. checkout sources 2. setup python 3. setup conda 4. keep order after set platform variables * change macos back to 12 since also fails with 11 * update condition in run the tests make difference between main or not main * fix path to cache invokeai conda env * fix invokeai conda env cache path * update mkdocs-flow.yml * change conda-channel priority * update create-caches * update conda env also when cache was used * os dependend conda env cache path * use existing CONDA env pointing to conda root * create CONDA_ROOT output from $CONDA * use output variable to define test prompts * use setup-python v4, get rid of PYTHON_BIN env * add runner.os to result artifacts name * update test-invoke-conda.yml: - reuse macos-latest - disable setup python 3.9 - setup conda with default python version - create or update conda environment depending on cache success - remove name parameter from conda update since name is set in env yml * improve mkdocs-flow.yml * disable cache-hugginface-torch since preload_models.py downloads to more than one location * update mkdocs-flow.yml with new name * rename mkdocs action to mkdocs-material * try to ignore error when creating conda env maybe it would still be usable, lets see ;P * remove bloat * update environment-mac.yml to match dependencies of invoke-ai/InvokeAI's main branch * disable conda update, tweak prompt condition * try to set some env vars for macOS to fix conda * stop ignoring error, use env instead of outputs * tweak `[[` connditions * update python and pip dependencie makes a difference of 1 sec per itteration compared to 3.9!!! also I see no reason why using a old pip version would be beneficial * remove unecesarry env for macOS everything was pre-tested on my MacBook Air 2020 with M1 * update conda env in setup step * activate conda env after installation * update test-invoke-conda.yml - set conda env dependent on matrix.os - set CONDA_ENV_NAME to prevent breaking action when renaming conda env - fix conda env activation * fix activate conda env * set bash -l as default shell * use action to activate conda env * add conda env file to env activation * try to replace s-weigeand with conda-incubator * remove azure-pipelines.yml funniest part is that the macos runner is the same as the one on github! * include environment-file in matrix - also disable auto-activate-base and auto-update-conda - include macos-latest and macos-12 for debugging purpose - set miniforge-version in matrix * fix miniforge-variant, set fail-fast to false * add step to setup miniconda - make default shell a matrix variable - remove bloat * use a mac env yml without pinned versions * unpin nomkl, pytorch and torchvision also removed opencv-pyhton * cache conda pkgs dir instead of conda env * use python 3.10, exclude macos-12 from cache * fix expression * prepare for PR * fix doubled id * reuse pinned versions in mac conda env - updated python pip version - unpined pytorch and torchvision - removed opencv-python - updated versions to most recent (tested locally) * fix classical copy/paste error * remove unused env from shell-block comment * fix hashFiles function to determine restore-keys * reenable caching `~.cache`, update create-caches * unpin all versions in mac conda env file this was the only way I got it working in the action, also works locally tested on MacBook Air 2020 M1 remove environment-mac-unpinned.yml * prepare merge by removing this branch from trigger * include pull_request trigger for main and dev * remove pull_request trigger
@lstein @tildebyte @Any-Winter-4079 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAICT, 'create-caches.yml' has a couple of issues:
- "Use Cached Stable Diffusion v1.4 Model" is duplicated
- "Use cached Conda Environment" and "Use cached Conda Packages" look to be almost exactly the same
Also, please unpin 'environment.yml'
* fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * remove 4gotten trigger * Update-gh-actions (#6) * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * use macos-latest * try to update conda before creating mac env * better conda update trial * re-pin streamlit version * re-added trigger to run workflow in current branch * try to find out if conda mac env could be updated * install cmake, protobuf and rust b4 conda * add yes to conda update * lets try anaconda3-2022.05 * try environment.yml for mac as well * reenable conda mac env, add pip install also fix gitignore by changing from dream to invoke * remove - unecesary virtualenv creation - conda update change != macos back to == linux * remove cmake from brew install since pre-installed * disable opencv-python pip requirement * fixed commands to find latest package versions * update requirements for mac env * back to the roots - only install conda env depending on runner_os with or without extra env variables * check out macOS in azure-pipelines since becoming kind of tired of the GitHub Runner which is broken as ... * let's try to setup python and update conda env * initialize conda before using it * add trigger in azure-pipelines.yml * And another go for update first .... * update azure-pipelines.yml - add caching - add checkpoint download - add paths to trigger and more * unquote checkpoint-url * fix chekpoint-url variable * mkdir before downloading model * set pr trigger to main, rename anaconda cache * unique cacheHitVariables * try to use macos-latest instead of macos-12 * update test-invoke-conda.yml: - remove unecesarry echo step - use s-weigand/setup-conda@v1 - remove conda update from install deps step since updated with action * update test-invoke-conda.yml: - rename conda env cache from ldm to invokeai - reorder steps: 1. checkout sources 2. setup python 3. setup conda 4. keep order after set platform variables * change macos back to 12 since also fails with 11 * update condition in run the tests make difference between main or not main * fix path to cache invokeai conda env * fix invokeai conda env cache path * update mkdocs-flow.yml * change conda-channel priority * update create-caches * update conda env also when cache was used * os dependend conda env cache path * use existing CONDA env pointing to conda root * create CONDA_ROOT output from $CONDA * use output variable to define test prompts * use setup-python v4, get rid of PYTHON_BIN env * add runner.os to result artifacts name * update test-invoke-conda.yml: - reuse macos-latest - disable setup python 3.9 - setup conda with default python version - create or update conda environment depending on cache success - remove name parameter from conda update since name is set in env yml * improve mkdocs-flow.yml * disable cache-hugginface-torch since preload_models.py downloads to more than one location * update mkdocs-flow.yml with new name * rename mkdocs action to mkdocs-material * try to ignore error when creating conda env maybe it would still be usable, lets see ;P * remove bloat * update environment-mac.yml to match dependencies of invoke-ai/InvokeAI's main branch * disable conda update, tweak prompt condition * try to set some env vars for macOS to fix conda * stop ignoring error, use env instead of outputs * tweak `[[` connditions * update python and pip dependencie makes a difference of 1 sec per itteration compared to 3.9!!! also I see no reason why using a old pip version would be beneficial * remove unecesarry env for macOS everything was pre-tested on my MacBook Air 2020 with M1 * update conda env in setup step * activate conda env after installation * update test-invoke-conda.yml - set conda env dependent on matrix.os - set CONDA_ENV_NAME to prevent breaking action when renaming conda env - fix conda env activation * fix activate conda env * set bash -l as default shell * use action to activate conda env * add conda env file to env activation * try to replace s-weigeand with conda-incubator * remove azure-pipelines.yml funniest part is that the macos runner is the same as the one on github! * include environment-file in matrix - also disable auto-activate-base and auto-update-conda - include macos-latest and macos-12 for debugging purpose - set miniforge-version in matrix * fix miniforge-variant, set fail-fast to false * add step to setup miniconda - make default shell a matrix variable - remove bloat * use a mac env yml without pinned versions * unpin nomkl, pytorch and torchvision also removed opencv-pyhton * cache conda pkgs dir instead of conda env * use python 3.10, exclude macos-12 from cache * fix expression * prepare for PR * fix doubled id * reuse pinned versions in mac conda env - updated python pip version - unpined pytorch and torchvision - removed opencv-python - updated versions to most recent (tested locally) * fix classical copy/paste error * remove unused env from shell-block comment * fix hashFiles function to determine restore-keys * reenable caching `~.cache`, update create-caches * unpin all versions in mac conda env file this was the only way I got it working in the action, also works locally tested on MacBook Air 2020 M1 remove environment-mac-unpinned.yml * prepare merge by removing this branch from trigger * include pull_request trigger for main and dev * remove pull_request trigger
54f5547
to
8a6ba6a
Compare
May I ask what's the purpose behind "create-caches.yml" at all? maybe we could even delete it completeley, going to unpin versions in environment.yml then, but since it had no problems (even works on Linux and Windows) and version pinning is a good thing in the end (when it is working for everybody ofc), I would have left them pinned until further solution is worked out. |
as asked by @tildebyte
Conda is actually right, it is caching Environment and Packages
Environment caching is not best practice, but since I thought it would run more than it will do now, it made totally sence (since conda pkgs/environment won't change often, but Code would). |
@tildebyte to come back to the "bare minimal test", imho the test which is running is pretty huge: all but refs/heads/main:
refs/heads/main:
this does a lot (but is fine to make sure it is still working ofc, but I also agree that for validating a PR this is pure overkill) This can be changed by editing the two files in tests/*prompts.txt, also can be advanced ofc |
What - how can I remove a requested review xD Sorry, did only pres the "Re-require review" button to point out that the asked changes are implemented 🙆🏻♂️ |
@tildebyte this would not have been pointed out without PR-Validation
needed to revert this since unpinned environment yml failed: Installing pip dependencies: ...working... ::warning::Pip subprocess error:%0AERROR: Could not detect requirement name for 'git+https://github.com/openai/CLIP.git@main#egg', please specify one with #egg=your_package_name%0A%0A
Pip subprocess error:
ERROR: Could not detect requirement name for 'git+https://github.com/openai/CLIP.git@main#egg', please specify one with #egg=your_package_name
Ran pip subprocess with arguments:
['/usr/share/miniconda3/envs/invokeai/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/runner/work/InvokeAI/InvokeAI/condaenv.6u8ex5u2.requirements.txt']
Pip subprocess output:
failed
Warning:
CondaEnvException: Pip failed |
also want to show how much faster it will run now with cached pkgs
So atm we have:
which imho should be totally worth the time vs merging something which then breaks buildability, reverting this, .... @tildebyte @lstein @Any-Winter-4079 What's your opinion? (not to mention that open source projects get the runtime 4 free 🤑) And to have done something "useful" to get a rerun with caches I just included a step to list current conda packages (since a pip freeze > ...txt gives me something not very helpful (less pkg-versions than installed locations...) |
should be ready to merge @tildebyte @lstein |
* fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * remove 4gotten trigger * Update-gh-actions (#6) * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * use macos-latest * try to update conda before creating mac env * better conda update trial * re-pin streamlit version * re-added trigger to run workflow in current branch * try to find out if conda mac env could be updated * install cmake, protobuf and rust b4 conda * add yes to conda update * lets try anaconda3-2022.05 * try environment.yml for mac as well * reenable conda mac env, add pip install also fix gitignore by changing from dream to invoke * remove - unecesary virtualenv creation - conda update change != macos back to == linux * remove cmake from brew install since pre-installed * disable opencv-python pip requirement * fixed commands to find latest package versions * update requirements for mac env * back to the roots - only install conda env depending on runner_os with or without extra env variables * check out macOS in azure-pipelines since becoming kind of tired of the GitHub Runner which is broken as ... * let's try to setup python and update conda env * initialize conda before using it * add trigger in azure-pipelines.yml * And another go for update first .... * update azure-pipelines.yml - add caching - add checkpoint download - add paths to trigger and more * unquote checkpoint-url * fix chekpoint-url variable * mkdir before downloading model * set pr trigger to main, rename anaconda cache * unique cacheHitVariables * try to use macos-latest instead of macos-12 * update test-invoke-conda.yml: - remove unecesarry echo step - use s-weigand/setup-conda@v1 - remove conda update from install deps step since updated with action * update test-invoke-conda.yml: - rename conda env cache from ldm to invokeai - reorder steps: 1. checkout sources 2. setup python 3. setup conda 4. keep order after set platform variables * change macos back to 12 since also fails with 11 * update condition in run the tests make difference between main or not main * fix path to cache invokeai conda env * fix invokeai conda env cache path * update mkdocs-flow.yml * change conda-channel priority * update create-caches * update conda env also when cache was used * os dependend conda env cache path * use existing CONDA env pointing to conda root * create CONDA_ROOT output from $CONDA * use output variable to define test prompts * use setup-python v4, get rid of PYTHON_BIN env * add runner.os to result artifacts name * update test-invoke-conda.yml: - reuse macos-latest - disable setup python 3.9 - setup conda with default python version - create or update conda environment depending on cache success - remove name parameter from conda update since name is set in env yml * improve mkdocs-flow.yml * disable cache-hugginface-torch since preload_models.py downloads to more than one location * update mkdocs-flow.yml with new name * rename mkdocs action to mkdocs-material * try to ignore error when creating conda env maybe it would still be usable, lets see ;P * remove bloat * update environment-mac.yml to match dependencies of invoke-ai/InvokeAI's main branch * disable conda update, tweak prompt condition * try to set some env vars for macOS to fix conda * stop ignoring error, use env instead of outputs * tweak `[[` connditions * update python and pip dependencie makes a difference of 1 sec per itteration compared to 3.9!!! also I see no reason why using a old pip version would be beneficial * remove unecesarry env for macOS everything was pre-tested on my MacBook Air 2020 with M1 * update conda env in setup step * activate conda env after installation * update test-invoke-conda.yml - set conda env dependent on matrix.os - set CONDA_ENV_NAME to prevent breaking action when renaming conda env - fix conda env activation * fix activate conda env * set bash -l as default shell * use action to activate conda env * add conda env file to env activation * try to replace s-weigeand with conda-incubator * remove azure-pipelines.yml funniest part is that the macos runner is the same as the one on github! * include environment-file in matrix - also disable auto-activate-base and auto-update-conda - include macos-latest and macos-12 for debugging purpose - set miniforge-version in matrix * fix miniforge-variant, set fail-fast to false * add step to setup miniconda - make default shell a matrix variable - remove bloat * use a mac env yml without pinned versions * unpin nomkl, pytorch and torchvision also removed opencv-pyhton * cache conda pkgs dir instead of conda env * use python 3.10, exclude macos-12 from cache * fix expression * prepare for PR * fix doubled id * reuse pinned versions in mac conda env - updated python pip version - unpined pytorch and torchvision - removed opencv-python - updated versions to most recent (tested locally) * fix classical copy/paste error * remove unused env from shell-block comment * fix hashFiles function to determine restore-keys * reenable caching `~.cache`, update create-caches * unpin all versions in mac conda env file this was the only way I got it working in the action, also works locally tested on MacBook Air 2020 M1 remove environment-mac-unpinned.yml * prepare merge by removing this branch from trigger * include pull_request trigger for main and dev * remove pull_request trigger
as asked by @tildebyte
@tildebyte this would not have been pointed out without PR-Validation
also want to show how much faster it will run now with cached pkgs
also switch to python between 3.9-3.10 and use conda-forge again
also change content of hashFile-function
remove restore-keys, make keys uniuqe
I hope I did the rebase correct now 🙈 |
in favor of #1118 🙈