[JUnit] Invoke (Before|After)Class and TestRules around Cucumber execution

[mpkorstanje]

Junit invokes several framework methods around the test execution.
The cucumber runner should finish its whole execution inside these.

To be precises these two should be equivalent:

@BeforeClass
public static void before {
   // Do stuff
}

@Test
public void test(){
    cucumber.api.cli.Main.main(some arguments....)
}

@AfterClass
public static void after {
   // Do stuff
}

@RunWith(Cucumber.class)
public class Test {
    @BeforeClass
    public static void before {
       // Do stuff
    }

    @AfterClass
    public static void after {
     // Do stuff
    }
}

By firing the TestRunFinished event and printing the summary directly
after invoking the children rather then after the completion of the
whole test run these are made equivalent again.

Firing the TestRunFinished event before the JUnit execution
is done might look a bit odd but semantically this is a cucumber event
not a junit event and cucumber has finished its execution at this point.

Context

https://cucumberbdd.slack.com/archives/C590XDQQH/p1501620355467970

Types of changes

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist:

• I've added tests for my code.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• Awaiting confirmation from toothpicks131

I've manually placed breakpoints in the examples above to verify the methods are called in the correct order.

[brasmusson]

I can see how this makes sense. There are however some interesting differences in the fine details of execution depending on which runner that is used, for instance:

• The JUnit and TestNG runners will establish the test count to be executed before any scenario is executed (because it needs to be communicated to the JUnit/TestNG infrastructure), but the CLI runner will compile and execute each feature file in sequence (so the test count from the later feature files are not known when the scenarios from the first feature file is executed).
• The TestNG runner will finish the Cucumber execution (send the TestRunFinished event and print the summary) in a @AfterClass method, whereas the JUnit runner with this PR will have done that before any @AfterClass method has been executed.

From the Slack thread triggering this PR:

The challenge, though, is that the cucumber reports are not available when junit's AfterClass hook executes.

In this Slack thread the proposed @beforeall/@afterall are also discussed. However since @beforeall/@afterall (if implemented) will be part of the Cucumber execution, they would not be equivalent to JUnit @BeforeClass/@AfterClass methods. Specifically the Started/Finished events for these hooks should be available to the formatters and event listeners (which I have argued for Cucumber-Ruby, so the cucumber report cannot be available when a @afterall hook is executed.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.