Parallel Execution of Tests

[darasandeep91]

👓 What did you see?

When Running Cucumber Tests with parallel strategy as fixed and parallelism set to 4. I am seeing more than 8 tests running in parallel.

✅ What did you expect to see?

I should see only 4 tests to run in parallel mode.

📦 Which tool/library version are you using?

I am using cucumber-junit-platform-engine with version 7.2.3

🔬 How could we reproduce it?

we need to enable parallel execution and set the parallel execution strategy to fixed.

cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=fixed
cucumber.execution.parallel.config.fixed.parallelism=4

Steps to reproduce the behavior:

1. Install '7.2.3' version of cucumber, cucumber-junit-platform-engine using the pom file

 <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

2. Add cucumber tests
3. Add the config mentioned above to junit-platform.properties file

cucumber.plugin=pretty
cucumber.glue=PATH_TO_STEP-Definitions
cucumber.filter.tags=@Smoke and not (@Ignore)
cucumber.publish.enabled=false

cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=fixed
cucumber.execution.parallel.config.fixed.parallelism=4

5. run the cucumber tests using runner file

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
public class SmokeTestRunnerTest {

}

📚 Any additional context?

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[mpkorstanje]

Parallel execution in the Cucumber JUnit Platform Engine is facilitated by JUnit 5.

cucumber-jvm/junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/CucumberTestEngine.java

Lines 47 to 55 in bb05e25

	@Override
	protected HierarchicalTestExecutorService createExecutorService(ExecutionRequest request) {
	ConfigurationParameters config = request.getConfigurationParameters();
	if (config.getBoolean(PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME).orElse(false)) {
	return new ForkJoinPoolHierarchicalTestExecutorService(
	new PrefixedConfigurationParameters(config, PARALLEL_CONFIG_PREFIX));
	}
	return super.createExecutorService(request);
	}

So this looks like an instance of junit-team/junit5#1858. Presumably this will be fixed in next version of JUnit with junit-team/junit5#2792. As the JUnit team uses engagement as a proxy for importance it would be worthwhile to verify and acknowledge this MR does indeed solve your problem and otherwise show interest by upvoting these issues.

As a workaround it may be possible to wrap whatever object creates the additional threadpool (e.g. a webdriver or kubernet client) in an object pool to ensure only a limited number of instances are ever created in total without depending on the exact number of threads.

[mpkorstanje]

Closing this. There is little to be done by Cucumber.

[fslev]

Use a custom parallel execution configuration strategy (don't forget to implement the new getSaturatedPredicate method - Junit 1.9.0)

https://github.com/fslev/cucumber-selenium-tutorial/blob/main/src/test/java/io/cucumber/selenium/tutorial/config/FixedParallelExecutionConfigurationStrategy.java

junit-platform.properties

cucumber.execution.parallel.config.custom.class=io.cucumber.selenium.tutorial.config.FixedParallelExecutionConfigurationStrategy

This way, you will always have a fixed number of threads running your scenarios.

[mpkorstanje]

Should probably add that to the docs.

[thomasrapadmi]

Hi guys, I've just run into the same issue here with all tests running at once even after setting the max thread to 4. I was hoping that I might try @fslev's solution but to no avail. Would it be possible to get a bit more information on how to integrate this custom parallel class with my tests as it seems to have no effect when adding it my project or might this be added to the docs anytime soon? Also would you be opposed to the idea of reverting to Junit4 to get the parallel execution working, if this is an issue with Junit5?

[mpkorstanje]

Would it be possible to get a bit more information on how to integrate this custom parallel class with my tests as it seems to have no effect when adding it my project or might this be added to the docs anytime soon?

You would have to set in junit-platform.properties:

cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=custom
cucumber.execution.parallel.config.custom.class=com.example.MyCustomParallelStrategy

See: https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#configuration-options

This follows the same pattern as JUnit Jupiter: https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution-config.

Also would you be opposed to the idea of reverting to Junit4 to get the parallel execution working, if this is an issue with Junit5?

This doesn't quite parse. Currently you can use either the cucumber-junit module for JUnit 4, or the cucumber-junit-platform-engine module for JUnit 5.

[thomasrapadmi]

I appreciate the response although that doesn't really help much, as I said I've already tried this. I set it in the junit-platform.properties file already exactly as your example but that does not work. On the second point, I think I may have formatted second question a little better, I meant would it be to my detriment to start using JUnit 4 as obviously JUnit 5 is the preferred option and I'm unsure of any drawbacks or issue that might arise from downgrading to JUnit 4.

[mpkorstanje]

Can you provide a bit more information about what appears to be broken when a strategy with the saturate predicate is provided? Either that or provide an MCVE.

I'm unsure of any drawbacks or issue that might arise from downgrading to JUnit 4.

On top of mind, JUnit 4 biggest draw back with respect to parallel execution is that it is limited to executing feature files rather then scenarios in parallel and because of that not very efficient. Other then that I can't possibly make any statements about what problems you may encounter in your context.

[mpkorstanje]

@thomasrapadmi I think I may have found your problem somewhat by accident. If you are on Java 8, it is not possible to configure the max pool size of the fork join pool used by JUnit. So the custom strategy will not work. Consider upgrading your Java version.

[fslev]

Also, a new Junit version (5.10.0) will soon be available, which uses saturated thread pool by default:
https://junit.org/junit5/docs/snapshot/release-notes/#deprecations-and-breaking-changes-2
Hence, no need for a custom parallel execution configuration strategy anymore, in this case.
Thus, the
cucumber.execution.parallel.config.strategy=fixed should work as expected.