Skip to content

Support for fail-fast in Ordered tests #3099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
abelsromero opened this issue Dec 1, 2022 · 2 comments
Closed

Support for fail-fast in Ordered tests #3099

abelsromero opened this issue Dec 1, 2022 · 2 comments

Comments

@abelsromero
Copy link

We use @Order to define user stories so that each test in a class accomplishes a validation and strictly depends on previous ones and is required by the following ones.
Currently @Order in case of error continues executing the following tests in the class which causes other tests to fail consuming resources and polluting test reports.

I'd be useful to have a fail-fast configuration so that on a failure no more tests are executed.
This would be specially helpful in convination of Test Retry Gradle plugin with new classRetry feature.

Example:
Currently:
on a class with tests 1, 2 and 3. If two fails we see 1, 2 and 3 run, with 2 and 3 failing.

We'd expect that:
on a class with tests 1, 2 and 3. If two fails we see 1, 2, with 2 failing. 3 should be ignored or marked as skipped.

@marcphilipp
Copy link
Member

marcphilipp commented Dec 1, 2022

It's possible to implement this behavior using a custom extension:

@SkipRemainingOnFailure
class SkipRemainingOnFailureDemo {

	@Test
	@Order(1)
	void a() {
	}

	@Test
	@Order(2)
	void b() {
		fail();
	}

	@Test
	@Order(3)
	void c() {
	}
}
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@TestMethodOrder(OrderAnnotation.class)
@ExtendWith(SkipRemainingOnFailure.Extension.class)
@interface SkipRemainingOnFailure {

	class Extension implements TestWatcher, ExecutionCondition {

		static final Namespace NAMESPACE = Namespace.create(Extension.class);

		@Override
		public void testFailed(ExtensionContext context, Throwable cause) {
			// Store in extension context of parent (i.e. test class) so that subsequent test can see failure
			context.getParent()
					.orElseThrow(() -> new RuntimeException("test without parent"))
					.getStore(NAMESPACE)
					.put("failed", true);
		}

		@Override
		public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
			return context.getStore(NAMESPACE).getOrDefault("failed", Boolean.class, false)
					? ConditionEvaluationResult.disabled("Skipping due to prior failure")
					: ConditionEvaluationResult.enabled("No prior failure");
		}
	}
}

@marcphilipp
Copy link
Member

Duplicate of #48

@marcphilipp marcphilipp marked this as a duplicate of #48 Dec 1, 2022
@marcphilipp marcphilipp closed this as not planned Won't fix, can't repro, duplicate, stale Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants