Create fail_over Option

[krisdestruction]

We support --cov-fail-under and fail_under options to fail tests when tests are under some coverage amount. #141

In an effort to asymptotically get to 100%, some repos may want --cov-fail-over and fail_over options to ensure the fail_under config is updated.

[blueyed]

some repos may want --cov-fail-over and fail_over options to ensure the fail_under config is updated.

Really? So that you get the extra work of updating the lower limit in case you manage to improve coverage just a bit too much?

I rather suggest using services that a) ensure that the diff is covered enough, and b) that coverage does not drop in general, but only increases.

Basically you should use the current/previous percentage for fail_under always. This could also be done by storing it somewhere as an artifact on CI, when not using services like codecov that manage it for you.

[krisdestruction]

Yes really, because working with legacy systems with lower coverage means that there's lots of room to improve. Can you give an example of such a service you're suggesting?

As for my feature request, I can't imagine a fail_over would be too code divergent than a fail_under

[blueyed]

I suggest using https://codecov.io/ - it can provide status about the patch being covered, and also the project's coverage not dropping.
It defaults to having the patch covered by the amount of current coverage (IIRC), but can be made to check for it to be covered by e.g. 100%.

[ionelmc]

I think it's an opportune moment to start working on a hookspec so people can implement any checks like fail over and stop the endless process of adding cli options.

…

On Fri, Aug 16, 2019, 18:07 Daniel Hahler ***@***.***> wrote: I suggest using https://codecov.io/ - it can provide status about the patch being covered, and also the project's coverage not dropping. It defaults to having the patch covered by the amount of current coverage (IIRC), but can be made to check for it to be covered by e.g. 100%. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#320?email_source=notifications&email_token=AAA7TXLTO7COEOGRNTB4WPDQE27ENA5CNFSM4ILZF2L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4O34BI#issuecomment-522042885>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA7TXMNFMXR5ZGSBL4B3ITQE27ENANCNFSM4ILZF2LQ> .

[nedbat]

I suggest there's no need to add this to pytest-cov at all. Let pytest run the tests. Let coverage report on coverage. See #337.