Potential CI improvement for multi-distro testing

[rgommers]

We now use the Sage CI jobs in https://github.com/FFY00/meson-python/blob/main/.github/workflows/ci-sage.yml. Those take quite a while:

I recently noticed how Meson does this:

• https://github.com/mesonbuild/meson/blob/7912901accaee714fc86febdc72f4347b9397759/.github/workflows/os_comp.yml#L40
• https://github.com/mesonbuild/meson/tree/7912901accaee714fc86febdc72f4347b9397759/ci/ciimage

That takes just over 1 minute to initialize a container, and then the tests start running. Given how faster the meson-python test suite is, doing it that way should be ~10x faster than through rebuilding Sage Docker image. I'm not sure if we'd lose anything in terms of coverage of configurations (@mkoeppe please comment if I'm missing something important here).

And we could use that extra time to test:

• In a venv
• With the distro Python
• Using --user
• Using sudo

[FFY00]

As far as I understand, what happens there is that the container image is built once and reused. The drawback here would be that then the image might contain older versions of stuff, which we will probably need a way to deal with.

But yeah, I agree this would be great.

[rgommers]

We could just rebuild the Docker images themselves weekly or monthly. Base OS images don't change a lot.

[FFY00]

Yeah, that makes sense. We can have a scheduled CI job to do that.

[mkoeppe]

I'm not sure if we'd lose anything in terms of coverage of configurations

Looking at https://github.com/mesonbuild/meson/blob/7912901accaee714fc86febdc72f4347b9397759/.github/workflows/os_comp.yml, I'd say they are complementary. It only has ubuntu and arch, but it has some compiler configs and some cross compiling set up, which is nice. ci-sage has other platforms such as fedora, gentoo, ...

[mkoeppe]

If speed is a concern for ci-sage, yes, incremental builds are a possibility.
We also have pre-built Docker images from which we could seed the build - https://github.com/orgs/sagemath/packages?repo_name=sage

[mkoeppe]

#183 makes ci-sage a bit faster. It's still a from-scratch build, but using a much smaller configuration of system packages. Incremental builds will be easier to do after https://trac.sagemath.org/ticket/34081 has been merged

[eli-schwartz]

As far as I understand, what happens there is that the container image is built once and reused. The drawback here would be that then the image might contain older versions of stuff, which we will probably need a way to deal with.

Meson has a weekly image builder job that attempts to build the container image from scratch, run tests, and on success pushes the newly built image to the docker hub. So older versions of stuff do get bumped, just not on every build.

This also means that we usually avoid dependency updates if those dependencies break the CI, so PRs only go red if the PR itself introduces a problem. This has its advantages and disadvantages. Jussi gets an email if the CI images are failing to build. On the other hand, people may not always look at it. It might be neat if the workflow could somehow submit or update an open issue on failures.

Actually there are two broken images, but I am fixing them as we speak...
(The third issue is just a bug in docker, apparently it takes Github ridiculous amounts of time to update literally anything...)

[dnicolodi]

For testing the macOS arm64 wheels I looked into Cirrus CI. They have a very nice service where given a Dockerfile they take care of the image build and caching. I tried instantiating a simple Debian unstable image with python and basic development tools, enough to build a python extension module. Building the image takes around 90 seconds. However, after the first build the image is cached and instantiated almost instantaneously. This seems a solid and easy to setup alternative to the Sage test infrastructure that may be better tailored to the needs of meson-python. This would also move some of the jobs from the busy GitHub Actions workers to another service with pretty high usage limits. If there is interest I can try to prepare a PR with a setup that runs the tests on a couple of different distributions and we can see how we like it.

Here is a minimal example https://github.com/dnicolodi/cirrusci

[rgommers]

That looks super clean. I've been quite impressed with Cirrus CI so far. +1 for giving this a try for both arm64 and multi-distro testing.

[henryiii]

FWIW we’ve been happy with Cirrus in cibuildwheel.

[dnicolodi]

Implemented in #241