Skip to content

Unable to use surefire maven plugin to rerun failing tests #1120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mpkorstanje opened this issue May 10, 2017 · 1 comment · Fixed by #1134
Closed

Unable to use surefire maven plugin to rerun failing tests #1120

mpkorstanje opened this issue May 10, 2017 · 1 comment · Fixed by #1134

Comments

@mpkorstanje
Copy link
Contributor

mpkorstanje commented May 10, 2017 •
edited
Loading

Summary

Using surefire to rerun failing tests results in a stack trace.

  • cucumber-java8:1.2.5
  • maven-failsafe-plugin:2.20
  • maven-surefire-plugin:2.20

Expected Behavior

When using surefire with mvn -Dfailsafe.rerunFailingTestsCount=2 test I expect the failing tests to be rerun.

Current Behavior

org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test class 'Scenario: Fail when running'
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)

Possible Solution

Surefire parses the test description for the class to rerun and expects a class name here rather then a human readable name. Either provide a class name or help convince the surefire folks to fix this this. :)

See: SUREFIRE-1372

Steps to Reproduce (for bugs)

  1. Setup mvn project with a failing cucumber test
  2. Add a test runner that uses @RunWith(Cucumber.class) and @CucumberOptions(...)
  3. Run mvn -Dfailsafe.rerunFailingTestsCount=2 test
  4. Observe stacktrace.

Context & Motivation

Rerunning failing tests as an integral part of the build, especially in large tests sets that are ran overnight, allows us to prioritize investigating the genuine failures over the flaky ones.
This was referenced May 10, 2017
Closed
Closed
mpkorstanje added a commit that referenced this issue May 25, 2017
@mpkorstanje
Rerunning failed tests with Surefire requires a method to identify th…
e4ca628 
…e tests.

Surefire currently uses a tuple of (Class, Method) to identify failed tests.
Cucumber-jvm uses test suites which do not consist of Classes or methods.
Therefore another method is needed.

JUnit provides the option to select tests that match a Description.
Descriptions are compared using  a unique Id. To identify tests between
different runs we should provide Descriptions with a predictable unique
id.

The current implementation did not suffice. While the provided cucumber
elements are unique, they are not predictable between runs making it
impossible to use their descriptions to rerun a failed test.

After this change it will be possible to update Surefire to use
descriptions to rerun failed tests.

  Related Issues:
  - https://issues.apache.org/jira/browse/SUREFIRE-1372
  - #1120
  - temyers/cucumber-jvm-parallel-plugin#31

(cherry picked from commit 4cc4439)
mpkorstanje added a commit that referenced this issue May 25, 2017
@mpkorstanje
Use deterministic unique ids in Descriptions
2eb9c47 
Rerunning failed tests with Surefire requires a method to identify the tests.
Surefire currently uses a tuple of (Class, Method) to identify failed tests.
Cucumber-jvm uses test suites which do not consist of Classes or methods.
Therefore another method is needed.

JUnit provides the option to select tests that match a Description.
Descriptions are compared using an unique Id. To identify tests between
different runs we should provide Descriptions with a predictable unique
id.

The current implementation did not suffice. While the provided cucumber
elements are unique, they are not predictable. Each run would create a new
instance of the identifier making it impossible to use their descriptions
to rerun a failed test.

After this change it will be possible to update Surefire to use
descriptions to rerun failed tests.

  Related Issues:
  - https://issues.apache.org/jira/browse/SUREFIRE-1372
  - #1120
  - temyers/cucumber-jvm-parallel-plugin#31
@mpkorstanje mpkorstanje mentioned this issue May 25, 2017
Merged
5 tasks
mpkorstanje added a commit that referenced this issue May 26, 2017
@mpkorstanje
Use deterministic unique ids in Descriptions
782d69d 
Rerunning failed tests with Surefire requires a method to identify the tests.
Surefire currently uses a tuple of (Class, Method) to identify failed tests.
Cucumber-jvm uses test suites which do not consist of Classes or methods.
Therefore another method is needed.

JUnit provides the option to select tests that match a Description.
Descriptions are compared using an unique Id. To identify tests between
different runs we should provide Descriptions with a predictable unique
id.

The current implementation did not suffice. While the provided cucumber
elements are unique, they are not predictable. Each run would create a new
instance of the identifier making it impossible to use their descriptions
to rerun a failed test.

After this change it will be possible to update Surefire to use
descriptions to rerun failed tests.

  Related Issues:
  - https://issues.apache.org/jira/browse/SUREFIRE-1372
  - #1120
  - temyers/cucumber-jvm-parallel-plugin#31
mpkorstanje added a commit that referenced this issue May 26, 2017
@mpkorstanje
[JUnit] Use deterministic unique ids in Descriptions
19774fc 
Rerunning failed tests with Surefire requires a method to identify the tests.
Surefire currently uses a tuple of (Class, Method) to identify failed tests.
Cucumber-jvm uses test suites which do not consist of Classes or methods.
Therefore another method is needed.

JUnit provides the option to select tests that match a Description.
Descriptions are compared using an unique Id. To identify tests between
different runs we should provide Descriptions with a predictable unique
id.

The current implementation did not suffice. While the provided cucumber
elements are unique, they are not predictable. Each run would create a new
instance of the identifier making it impossible to use their descriptions
to rerun a failed test.

After this change it will be possible to update Surefire to use
descriptions to rerun failed tests.

  Related Issues:
  - https://issues.apache.org/jira/browse/SUREFIRE-1372
  - #1120
  - temyers/cucumber-jvm-parallel-plugin#31
@mpkorstanje mpkorstanje mentioned this issue May 26, 2017
Closed
@cdiminutto cdiminutto mentioned this issue Aug 9, 2017
Closed
@mpkorstanje mpkorstanje mentioned this issue May 4, 2018
Closed
@lock
Copy link

lock bot commented Oct 25, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Oct 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use deterministic unique ids in Descriptions cucumber/cucumber-jvm
1 participant
@mpkorstanje