Skip --cover when --max-failures is reached

[henriqueberlesi]

Elixir and Erlang/OTP versions

Erlang/OTP 25 [erts-13.2.2.12] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]

Elixir 1.18.1 (compiled with Erlang/OTP 25)

Operating system

Ubuntu 22.04.5 LTS

Current behavior

I've recently added --max-failures 1 to the following command:
MIX_TEST_PARTITION=1 mix test --cover --warnings-as-errors --raise --partitions 3 --max-failures 1

The current behavior when the test fails is to stop the suite (as intended), but the coverage report still runs after the abortion. This makes the coverage report incomplete (only showing 0.9% coverage when ~44% was expected for this partition).

Stacktrace:

Run MIX_TEST_PARTITION=1 mix test --cover --warnings-as-errors --raise --partitions 3 --max-failures 1
Running ExUnit with seed: 646065, max_cases: 1
Excluding tags: [:integration_tests]

.....................................

  1) test foobar should return the foobar when some condition is met (MyApp.FooBarTest)
Error:      test/my_app/foo_bar_test.exs:10
     ** (KeyError) key :foobar not found in: nil

     If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
     code: response = post(conn, "/graphql", query: @foobarquery) |> json_response(200)
     stacktrace:
       (absinthe 1.7.9) lib/absinthe/resolution.ex:206: Absinthe.Resolution.call/2

--max-failures reached, aborting test suite
Finished in 23.0 seconds (23.0s async, 0.00s sync)
38 tests, 1 failure
----------------
COV    FILE                                                                                                    LINES RELEVANT   MISSED
  0.0% test/support/utils/foo.ex                                                                                  53       10       10
  0.0% test/support/utils/bar.ex                                                                                   9        1        1
  0.0% test/support/utils/baz.ex                                                                                   5        1        1
  0.0% test/support/utils/qux/alpha.ex                                                                            99       27       27
  0.0% test/support/utils/qux/beta.ex                                                                            591      137      137
[TOTAL]   0.9%
----------------
Coverage data exported.
** (Mix) "mix test" failed
Error: Process completed with exit code 1.

Expected behavior

The expected behavior was for the coverage analysis report to only run when the test suite/partition is not aborted, since I'm trying to stop the test suite at the first sign of failure.

When running the coverage report on an aborted test partition, coverage metrics would be incomplete, possibly making them irrelevant at that stage. Also, with long coverage report (2k~ lines), I'm trying to avoid printing this report in the console if possible.

[henriqueberlesi]

Just edited the issue to add a few more details.

Not sure if this behavior makes sense but I'd be happy to provide more information if needed or to attempt to enable it if accepted.

[josevalim]

I am afraid this won't be trivial. From the point of view of coverage, the failure due to max_failures is the same as any other test suite failure, because those concerns are handled at two different levels. Is there a particular reason why you are requesting coverage when you know you may not go over the whole suite?

[henriqueberlesi]

I see. This command is used in a CI check, which is required to pass before merges. The coverage report is generated after each partition run and then compiled using mix test.coverage.

The rationale behind adding the max-failures flag was to terminate the execution of the check as soon as possible (for example, in the scenario of a flaky test). Given that, I expected the coverage report not run, since the CI would need to be re-run anyway.

However, this is a minor issue for this specific scenario and not a blocker whatsoever. If it's not trivial and would add unnecessary coupling between two levels of the system, I’m certain there are other ways to achieve what I’m aiming for.

Thanks for the response!

[josevalim]

Thank you. I will go ahead and close this then, as it is indeed non-straight-forward.