Stream closed exception while trying to finish TestNGCucumberRunner

[fslev]

Hi,

I have the a TestNG class for running Cucumber scenarios both serially and in parallel.
It contains two @test methods: one is for running scenarios in parallel and the 2nd is for running scenarios in serial.
Test methods are executed in parallel:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="CucumberTutorialTestNgTestSuite" parallel="methods" data-provider-thread-count="8">
    <test name="AllTests">
        <packages>
            <package name="com.cucumber.tutorial.*"/>
        </packages>
    </test>
</suite>

TestNG class:

@CucumberOptions(features = "src/test/resources/features",
        glue = {"com.cucumber.utils", "com.cucumber.tutorial"},
        plugin = {"pretty", "junit:output", "json:target/cucumber-report/report.json"}, tags = {"not @Ignore", "not @ignore"})
public class TutorialTest {

    private static final Predicate<Pickle> isSerial = pickle -> pickle.getTags().contains("@Serial")
            || pickle.getTags().contains("@serial");

    private Logger log = LogManager.getLogger();
    private TestNGCucumberRunner testNGCucumberRunner;

    @BeforeClass(alwaysRun = true)
    public void setUpClass() {
        testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
    }

    @Test(groups = "cucumber parallel", description = "Runs Cucumber Parallel Scenarios", dataProvider = "parallelScenarios")
    public void runParallelScenario(PickleWrapper pickleWrapper, FeatureWrapper featureWrapper) throws Throwable {
        log.info("Preparing Parallel scenario ---> {}", pickleWrapper.getPickle().getName());
        testNGCucumberRunner.runScenario(pickleWrapper.getPickle());
    }

    @Test(groups = "cucumber serial", description = "Runs Cucumber Scenarios in the Serial group", dataProvider = "serialScenarios")
    public void runSerialScenario(PickleWrapper pickleWrapper, FeatureWrapper featureWrapper) throws Throwable {
        log.info("Preparing Serial scenario ---> {}", pickleWrapper.getPickle().getName());
        testNGCucumberRunner.runScenario(pickleWrapper.getPickle());
    }

    @DataProvider(parallel = true)
    public Object[][] parallelScenarios() {
        if (testNGCucumberRunner == null) {
            return new Object[0][0];
        }
        return filter(testNGCucumberRunner.provideScenarios(), isSerial.negate());
    }

    @DataProvider
    public Object[][] serialScenarios() {
        if (testNGCucumberRunner == null) {
            return new Object[0][0];
        }
        return filter(testNGCucumberRunner.provideScenarios(), isSerial);
    }

    @AfterClass(alwaysRun = true)
    public void tearDownClass() {
        if (testNGCucumberRunner == null) {
            return;
        }
        testNGCucumberRunner.finish();
    }

    private Object[][] filter(Object[][] scenarios, Predicate<Pickle> accept) {
        return Arrays.stream(scenarios).filter(objects -> {
            PickleWrapper candidate = (PickleWrapper) objects[0];
            return accept.test(candidate.getPickle());
        }).toArray(Object[][]::new);
    }
}

When all scenarios finish, the @afterclass method is invoked in order to close the TestNGCucumberRunner. Then, the following error is thrown:

[ERROR] tearDownClass(com.cucumber.tutorial.TutorialTest)  Time elapsed: 0.721 s  <<< FAILURE!
io.cucumber.core.exception.CucumberException: Error while transforming.
        at com.cucumber.tutorial.TutorialTest.tearDownClass(TutorialTest.java:63)
Caused by: javax.xml.transform.TransformerException: 
java.lang.RuntimeException: org.xml.sax.SAXException: java.io.IOException: Stream closed
java.io.IOException: Stream closed
        at com.cucumber.tutorial.TutorialTest.tearDownClass(TutorialTest.java:63)
Caused by: java.lang.RuntimeException: 
org.xml.sax.SAXException: java.io.IOException: Stream closed
java.io.IOException: Stream closed
        at com.cucumber.tutorial.TutorialTest.tearDownClass(TutorialTest.java:63)

But sometimes, when I run the code again, I get a NPE:

[ERROR] tearDownClass(com.cucumber.tutorial.TutorialTest)  Time elapsed: 0.469 s  <<< FAILURE!
java.lang.NullPointerException
        at com.cucumber.tutorial.TutorialTest.tearDownClass(TutorialTest.java:63)

This is even stranger, because this is the only stacktrace I get.

Should I create two separate runners in this case, for each test method ?
The code from above is actually taken from the cucumber-testng module:
ScenariosInDifferentGroupsTest
Cheers

[fslev]

If I change testng.xml file to run @test methods in serial, the issue / issues from above never happen

[mpkorstanje]

How often is tearDownClass invoked? Sounds like the test run finished event is emitted twice.

[fslev]

Added some logs inside the tearDownClass() method.
It seems that is invoked only once.

[fslev]

In order to reproduce it easier, you can pull the code from here:

[email protected]:fslev/cucumber-utils-tutorial.git

and run it:

mvn clean test -Pprod -Dconcurrent -Dtags=@all

[mpkorstanje]

Thanks for the reproducer. Though I would have preferred a slightly smaller one. The additional output made it hard add see what was going on.

The problem is that while Cucumber can execute scenarios in parallel it can not execute data provider methods in parallel. Because you've set parallel="methods" TestNG will however try to do so. This will be fixed with #1919 but for the time being I would recommend that you don't try to execute data provider methods in parallel.

[fslev]

Thanks !