RFC: Parallelizing across machines in CI

[jamiebuilds]

Many CI systems support splitting up a single task across multiple machines. This can drastically speed up CI times even when you are duplicating some of the work compiling or whatever.

From what I've seen, they mostly work like this:

steps:
  - label: test
    commands:
      - yarn install
      - yarn test
    parallelism: 5

# on machine 1
CI_JOB=0 CI_JOB_COUNT=5 yarn install
CI_JOB=0 CI_JOB_COUNT=5 yarn test

# on machine 2
CI_JOB=1 CI_JOB_COUNT=5 yarn install
CI_JOB=1 CI_JOB_COUNT=5 yarn test

# on machine 3
CI_JOB=2 CI_JOB_COUNT=5 yarn install
CI_JOB=2 CI_JOB_COUNT=5 yarn test
...

For example: https://buildkite.com/docs/builds/parallel-builds

It would be awesome if there was an easy way to integrate this with Ava such that you could automatically chunk tests up so that they can be run across machines.

I imagine you'd have to discover all the tests and then have some stable way of splitting them up, either by counting the tests or by splitting the files up.

I think you could even automatically do this without any additional setup from developers. I created a module to help you do just that: https://github.com/jamiebuilds/ci-parallel-vars

[novemberborn]

Landed in #1812.