Cucumber should support the ability to gracefully abort a test suite

[andyRokit]

Is your feature request related to a problem? Please describe.
During tests suites, conditions may arise where it makes sense to stop the test suite.

E.g. While running set of integration tests, a downstream system has got into an invalid state.

Currently this is achievable by adding a System.exit to the test. E.g.

if(environmentInInvalidState()) {
  System.exit(1);
}

However, that effectively kills the JVM and prevents and other processes completing. E.g. Generating reports or cleaning up data.

I would suggest that a mechanism be added that allow a test to gracefully exit the suite.

Describe the solution you'd like
Add a new RunListener which listens for a specific exception coming from a test failure. If that exception is detected request that the test suite is abandoned using RunNotifier.pleaseStop().

This may be a new cucumber-specific exception or there may already be an exception which fits this purpose. E.g. StoppedByUserException

[mpkorstanje]

I don't see much reason to provide this in Cucumber. There are a few way to do this using only the JUnit 4 framework.

JUnit Suite

JUnit allows you to compose different runners into a suite. By extending the Suite class you can make your own runner for that Suite. This this will allow you request the suite to stop when receiving a test failure with a specific kind of error.

https://github.com/RaphC/selenese4j-maven-plugin/wiki/How-to-Stop-Junit-Tests-Suite-on-the-first-failure

Maven

Alternatively Maven lets you attach a RunListener directly. Again this will allow you request the suite to stop when receiving a test failure with a specific kind of error.

https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html#Using_Custom_Listeners_and_Reporters

[andyRokit]

Thanks @mpkorstanje - that makes sense. Closing issue.

[andyRokit]

@mpkorstanje - just a quick question about the Maven approach: Is it possible for the Listeners added in this way to get a handle on the RunNotifier (in order to request that the suite stop)?

[mpkorstanje]

Ah. You're right. Pretty sure there is a way because I have seen surefire call pleaseStop before. Maybe this? http://maven.apache.org/surefire/maven-surefire-plugin/examples/skip-after-failure.html

…

On Tue, Jan 7, 2020, 10:12 andyRokit ***@***.***> wrote: @mpkorstanje <https://github.com/mpkorstanje> - just a quick question about the Maven approach: Is it possible for the Listeners added in this way to get a handle on the RunNotifier (in order to request that the suite stop)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1854?email_source=notifications&email_token=ABVAAZ7CJUYXT7WJULU7XJTQ4RBOJA5CNFSM4KDFBV3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIIGRZY#issuecomment-571500775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVAAZZWRK4KP7DMSSSUPJLQ4RBOJANCNFSM4KDFBV3A> .

[andyRokit]

Yeah - I noticed that mechanism, but that'll stop the run on any failure(s). I'd want to stop the run on a specific event. I.e. "Downstream system X has got into an invalid state, no point in continuing".

Surefire doesn't appear to support adding any hooks on its listeners. It just initialises them and adds them.

Edit
Maybe this would be a good feature request for maven-surefire!

[mpkorstanje]

Yeah. The other option would be to use Assume from JUnit4 or Assumptions from JUnit5 to skip the scenarios. It won't abort the suite but it will mark the scenario as skipped. As long as you have one failing test the effect should be the same.

[andyRokit]

That's an interesting thought. Thanks for the insight - it's been really helpful.

[andyRokit]

Regarding the Maven Surefire option; I opened a feature request for maven surefire to see if it would make sense for them to provide the ability for their custom listeners to abort the test run.

[Tibor17]

It is an OSS project and so I expect the user's to develop this feature in Surefire. It is nothing new because this feature exists in three other providers and so it can be, so called, copy-paste in junit5 provider and it is quite feasible feature. Pls participate in the JIRA issue SUREFIRE-1743. If you need me to guide you regarding the development approach, feel free to ask me in JIRA.

[mpkorstanje]

@Tibor17 I didn't see a "please stop" implementation in JUnit 5. I did see you requested one. Would you happen to know if it has been implemented?

[Tibor17]

@mpkorstanje
I know and therefore see my comments in
https://issues.apache.org/jira/browse/SUREFIRE-1743?focusedCommentId=17010969&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17010969
and
Question: https://issues.apache.org/jira/browse/SUREFIRE-1698?focusedCommentId=16952731&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16952731
Answer: https://issues.apache.org/jira/browse/SUREFIRE-1698?focusedCommentId=16953638&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16953638

[mpkorstanje]

Ah thanks! I think it is possible for JUnit 5 to facilitate this but each TestEngine will have to do its own implementation. I might do proof of concept if I can find the time. For now I added a few notes to the issue.

junit-team/junit5#1880

[Tibor17]

The TestNG also does not have it implemented and we had to implement a similar extension in provider surefire-testng like you would do in junit5 provider surefire-junit-platform, see the usage in provider.
And we implemented our interface Stoppable and used it in TestNG listener called FailFastNotifier.
A kind of similar would be regarding junit5.