Skip to content

Commit 1eb282a

Browse files
projectgusdpgeorge
projectgus
authored and
dpgeorge
committed
tools/ci.sh: Support publishing package and index files to GitHub Pages.
Opt-in feature to make it easier for folks to test packages that are still in development, open in Pull Requests, or even in independent forks. --- To enable this on your own GitHub fork of the micropython-lib repository then navigate to the fork's "Settings" -> "Secrets and variables" -> "Actions" -> "Variables" page, then click "New repository variable", and create a variable named MIP_INDEX with value true (or any "truthy" value). Once enabled then any time a branch is pushed to your fork and builds successfully, GitHub Actions will also push the built packages and package index to the gh-pages branch which is associated with the repo's GitHub Pages web site. The packages can then be installed remotely via: mpremote mip --index \ https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME PACKAGE_NAME or on a device as: mip.install(PACKAGE_NAME, index="https://USERNAME.github.io/micropython-lib/mip/BRANCHNAME") (Replace USERNAME, BRANCH_NAME and PACKAGE_NAME as applicable. If you've renamed your fork, change the name micropython-lib to match.) Note: As well as the MIP_INDEX repository variable, this functionality depends on both GitHub Actions and GitHub Pages being enabled on your repository in GitHub. However both options should enable automatically, unless they have been manually disabled. This work was funded through GitHub Sponsors.
1 parent 9ee0257 commit 1eb282a

File tree

5 files changed

+196
-1
lines changed

5 files changed

+196
-1
lines changed

‎.github/workflows/build_packages.yml

+3
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,6 @@ jobs:
1414
run: source tools/ci.sh && ci_build_packages_check_manifest
1515
- name: Compile package index
1616
run: source tools/ci.sh && ci_build_packages_compile_index
17+
- name: Publish packages for branch
18+
if: vars.MICROPY_PUBLISH_MIP_INDEX && github.event_name == 'push' && ! github.event.deleted
19+
run: source tools/ci.sh && ci_push_package_index

‎.github/workflows/cleanup_published_packages.yml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
name: Cleanup published packages
2+
3+
on: delete
4+
5+
jobs:
6+
cleanup:
7+
runs-on: ubuntu-latest
8+
if: vars.MICROPY_PUBLISH_MIP_INDEX
9+
steps:
10+
- uses: actions/checkout@v2
11+
- name: Clean up published files
12+
run: source tools/ci.sh && ci_cleanup_package_index ${{ github.event.ref }}

‎CONTRIBUTING.md

+46
Original file line numberDiff line numberDiff line change
@@ -69,3 +69,49 @@ There are some specific conventions and guidelines for micropython-lib:
6969

7070
* When porting an existing third-party package, please ensure that the source
7171
license is compatible.
72+
73+
* To make it easier for others to install packages directly from your PR before
74+
it is merged, consider opting-in to automatic package publishing (see
75+
[Publishing packages from forks](#publishing-packages-from-forks)). If you do
76+
this, consider quoting the [commands to install
77+
packages](README.md#installing-packages-from-forks) in your Pull Request
78+
description.
79+
80+
### Publishing packages from forks
81+
82+
You can easily publish the packages from your micropython-lib
83+
[fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks)
84+
by opting in to a system based on [GitHub
85+
Actions](https://docs.github.com/en/actions) and [GitHub
86+
Pages](https://docs.github.com/en/pages):
87+
88+
1. Open your fork's repository in the GitHub web interface.
89+
2. Navigate to "Settings" -> "Secrets and variables" -> "Actions" -> "Variables".
90+
3. Click "New repository variable"
91+
4. Create a variable named `MICROPY_PUBLISH_MIP_INDEX` with value `true` (or any
92+
"truthy" value).
93+
5. The settings for GitHub Actions and GitHub Pages features should not need to
94+
be changed from the repository defaults, unless you've explicitly disabled
95+
them.
96+
97+
The next time you push commits to a branch in your fork, GitHub Actions will run
98+
an additional step in the "Build All Packages" workflow named "Publish Packages
99+
for branch".
100+
101+
Anyone can then install these packages as described under [Installing packages
102+
from forks](README.md#installing-packages-from-forks). The exact commands are also
103+
quoted in the GitHub Actions log for the "Publish Packages for branch" step.
104+
105+
#### Opting Back Out
106+
107+
To opt-out again, delete the `MICROPY_PUBLISH_MIP_INDEX` variable and
108+
(optionally) delete the `gh-pages` branch from your fork.
109+
110+
*Note*: While enabled, all micropython-lib packages will be published each time
111+
a change is pushed to any branch in your fork. A commit is added to the
112+
`gh-pages` branch each time. In a busy repository, the `gh-pages` branch may
113+
become quite large. The actual `.git` directory size on disk should still be
114+
quite small, as most of the content will be duplicated. If you're worried that
115+
the `gh-pages` branch has become too large then you can always delete this
116+
branch from GitHub. GitHub Actions will create a new `gh-pages` branch the next
117+
time you push a change.

‎README.md

+31
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,37 @@ Note that unlike the other three approaches based on `mip` or `manifest.py`,
8484
you will need to manually resolve dependencies. You can inspect the relevant
8585
`manifest.py` file to view the list of dependencies for a given package.
8686

87+
## Installing packages from forks
88+
89+
It is possible to use the `mpremote mip install` or `mip.install()` methods to
90+
install packages built from a
91+
[fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks)
92+
of micropython-lib, if the fork's owner has opted in.
93+
94+
This can be useful to install packages from a pending Pull Request, for example.
95+
96+
First, the owner of the fork must opt-in as described under
97+
[Publishing packages from forks](CONTRIBUTING.md#publishing-packages-from-forks).
98+
99+
After this has happened, each time someone pushes to a branch in that fork then
100+
GitHub Actions will automatically publish the packages to a GitHub Pages site.
101+
102+
To install these packages, use commands such as:
103+
104+
```bash
105+
$ mpremote connect /dev/ttyUSB0 mip install --index https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME PACKAGE_NAME
106+
```
107+
108+
Or from a networked device:
109+
110+
```py
111+
import mip
112+
mip.install(PACKAGE_NAME, index="https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME")
113+
```
114+
115+
(Where `USERNAME`, `BRANCH_NAME` and `PACKAGE_NAME` are replaced with the owner
116+
of the fork, the branch the packages were built from, and the package name.)
117+
87118
## Contributing
88119

89120
We use [GitHub Discussions](https://github.com/micropython/micropython/discussions)

‎tools/ci.sh

+104-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
#!/bin/bash
22

3+
########################################################################################
4+
# common "constants"
5+
PACKAGE_INDEX_PATH=/tmp/micropython-lib-deploy
6+
37
########################################################################################
48
# code formatting
59

@@ -38,5 +42,104 @@ function ci_build_packages_check_manifest {
3842
}
3943

4044
function ci_build_packages_compile_index {
41-
python3 tools/build.py --micropython /tmp/micropython --output /tmp/micropython-lib-deploy
45+
python3 tools/build.py --micropython /tmp/micropython --output $PACKAGE_INDEX_PATH
46+
}
47+
48+
function ci_push_package_index {
49+
set -euo pipefail
50+
51+
# Note: This feature is opt-in, so this function is only run by GitHub
52+
# Actions if the MICROPY_PUBLISH_MIP_INDEX repository variable is set to a
53+
# "truthy" value in the "Secrets and variables" -> "Actions"
54+
# -> "Variables" setting of the GitHub repo.
55+
56+
PAGES_PATH=/tmp/gh-pages
57+
58+
if git fetch --depth=1 origin gh-pages; then
59+
git worktree add ${PAGES_PATH} gh-pages
60+
cd ${PAGES_PATH}
61+
NEW_BRANCH=0
62+
else
63+
echo "Creating gh-pages branch for $GITHUB_REPOSITORY..."
64+
git worktree add --force ${PAGES_PATH} HEAD
65+
cd ${PAGES_PATH}
66+
git switch --orphan gh-pages
67+
NEW_BRANCH=1
68+
fi
69+
70+
DEST_PATH=${PAGES_PATH}/mip/${GITHUB_REF_NAME}
71+
if [ -d ${DEST_PATH} ]; then
72+
git rm -r ${DEST_PATH}
73+
fi
74+
mkdir -p ${DEST_PATH}
75+
cd ${DEST_PATH}
76+
77+
cp -r ${PACKAGE_INDEX_PATH}/* .
78+
79+
git add .
80+
git_bot_commit "Add CI built packages from commit ${GITHUB_SHA} of ${GITHUB_REF_NAME}"
81+
82+
if [ "$NEW_BRANCH" -eq 0 ]; then
83+
# A small race condition exists here if another CI job pushes to
84+
# gh-pages at the same time, but this narrows the race to the time
85+
# between these two commands.
86+
git pull --rebase origin gh-pages
87+
fi
88+
git push origin gh-pages
89+
90+
INDEX_URL="https://${GITHUB_REPOSITORY_OWNER}.github.io/$(echo ${GITHUB_REPOSITORY} | cut -d'/' -f2-)/mip/${GITHUB_REF_NAME}"
91+
92+
echo ""
93+
echo "--------------------------------------------------"
94+
echo "Uploaded package files to GitHub Pages."
95+
echo ""
96+
echo "Unless GitHub Pages is disabled on this repo, these files can be installed remotely with:"
97+
echo ""
98+
echo "mpremote mip install --index ${INDEX_URL} PACKAGE_NAME"
99+
echo ""
100+
echo "or on the device as:"
101+
echo ""
102+
echo "import mip"
103+
echo "mip.install(PACKAGE_NAME, index=\"${INDEX_URL}\")"
104+
}
105+
106+
function ci_cleanup_package_index()
107+
{
108+
if ! git fetch --depth=1 origin gh-pages; then
109+
exit 0
110+
fi
111+
112+
# Argument $1 is github.event.ref, passed in from workflow file.
113+
#
114+
# this value seems to be a REF_NAME, without heads/ or tags/ prefix. (Can't
115+
# use GITHUB_REF_NAME, this evaluates to the default branch.)
116+
DELETED_REF="$1"
117+
118+
if [ -z "$DELETED_REF" ]; then
119+
echo "Bad DELETE_REF $DELETED_REF"
120+
exit 1 # Internal error with ref format, better than removing all mip/ directory in a commit
121+
fi
122+
123+
# We need Actions to check out default branch and run tools/ci.sh, but then
124+
# we switch branches
125+
git switch gh-pages
126+
127+
echo "Removing any published packages for ${DELETED_REF}..."
128+
if [ -d mip/${DELETED_REF} ]; then
129+
git rm -r mip/${DELETED_REF}
130+
git_bot_commit "Remove CI built packages from deleted ${DELETED_REF}"
131+
git pull --rebase origin gh-pages
132+
git push origin gh-pages
133+
else
134+
echo "Nothing to remove."
135+
fi
136+
}
137+
138+
# Make a git commit with bot authorship
139+
# Argument $1 is the commit message
140+
function git_bot_commit {
141+
# Ref https://github.com/actions/checkout/discussions/479
142+
git config user.name 'github-actions[bot]'
143+
git config user.email 'github-actions[bot]@users.noreply.github.com'
144+
git commit -m "$1"
42145
}

0 commit comments

Comments
 (0)