Introduce a way to "print out" a ExUnit test plan

[whatyouhide]

I've had multiple occasions recently where I needed to know which tests ExUnit would run given some configuration. For example, if I run

MIX_TEST_PARTITION=5 mix test --partitions 6

or

mix test --only flaky

There's a "trick" to use which is to add --trace and go through the list of tests that run, but that's not very tool friendly.

I’m thinking we might introduce an API for printing out a test plan. It could be just that, without running the tests:

mix test.plan [flags]

or just printing out the plan before starting the suite:

mix test --print-plan [flags]

Thoughts?

[josevalim]

What about --dry-run? That effectively calls the formatters and what not, but does not effectively run the tests? It is as if all setup blocks and tests were a no-op?

[whatyouhide]

Mh that could be an option, but why would you want to do that other than for printing out the plan? In that case, why not call the option something more direct? (Just trying to understand, I think --dry-run would work too)

[josevalim]

Basically because --dry-run would compose with all other options. with --only, --seed, file filters, and whatever have you. Otherwise we would have to reimplement all option handling in a new task.

[whatyouhide]

@josevalim good point about the new task, scratch that. Also, I think it would be nice with --dry-run that it would work with custom formatters to print out the tests in any way a tool might want (without running them).

[josevalim]

@whatyouhide please open up a new issue with the description of what we want to go with (without the discussion) so others can contribute?

[whatyouhide]

Closing in favor of #14393.