Skip to content

Commit 5300e65

Browse files
sbrannensbrannen
committed
Introduce support for ordering the sequence of test methods
These commits introduce new support in JUnit Jupiter for ordering the sequence of test methods. Specifically, these commits introduce a new MethodOrderer API that can be registered via a new @TestMethodOrder annotation. JUnit Jupiter provides the following built-in implementations of this new API. - Alphanumeric: sorts test methods alphanumerically based on their names and formal parameter lists. - OrderAnnotation: sorts test methods numerically based on values specified via the @order annotation. - Random: orders test methods pseudo-randomly and supports configuration of a custom seed. Consult the new "Test Execution Order" section of the User Guide for details and example usage. Issue: #13
2 parents d8c4c90 + 7c1f77a commit 5300e65

File tree

17 files changed

+1311
-9
lines changed

17 files changed

+1311
-9
lines changed

‎documentation/src/docs/asciidoc/link-attributes.adoc

+6
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ endif::[]
3939
:AfterAllCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterAllCallback.html[AfterAllCallback]
4040
:AfterEachCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterEachCallback.html[AfterEachCallback]
4141
:AfterTestExecutionCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterTestExecutionCallback.html[AfterTestExecutionCallback]
42+
:Alphanumeric: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.Alphanumeric.html[Alphanumeric]
4243
:ArgumentsAccessor: {javadoc-root}/org/junit/jupiter/params/aggregator/ArgumentsAccessor.html[ArgumentsAccessor]
4344
:ArgumentsAggregator: {javadoc-root}/org/junit/jupiter/params/aggregator/ArgumentsAggregator.html[ArgumentsAggregator]
4445
:Assertions: {javadoc-root}/org/junit/jupiter/api/Assertions.html[org.junit.jupiter.api.Assertions]
@@ -62,10 +63,14 @@ endif::[]
6263
:ExtendWith: {javadoc-root}/org/junit/jupiter/api/extension/ExtendWith.html[@ExtendWith]
6364
:ExtensionContext: {javadoc-root}/org/junit/jupiter/api/extension/ExtensionContext.html[ExtensionContext]
6465
:ExtensionContext_Store: {javadoc-root}/org/junit/jupiter/api/extension/ExtensionContext.Store.html[Store]
66+
:MethodOrderer: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.html[MethodOrderer]
6567
:MethodSource: {javadoc-root}/org/junit/jupiter/params/provider/MethodSource.html[@MethodSource]
68+
:Order: {javadoc-root}/org/junit/jupiter/api/Order.html[@Order]
69+
:OrderAnnotation: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.OrderAnnotation.html[OrderAnnotation]
6670
:ParameterContext: {javadoc-root}/org/junit/jupiter/api/extension/ParameterContext.html[ParameterContext]
6771
:ParameterizedTest: {javadoc-root}/org/junit/jupiter/params/ParameterizedTest.html[@ParameterizedTest]
6872
:ParameterResolver: {javadoc-root}/org/junit/jupiter/api/extension/ParameterResolver.html[ParameterResolver]
73+
:Random: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.Random.html[Random]
6974
:RegisterExtension: {javadoc-root}/org/junit/jupiter/api/extension/RegisterExtension.html[@RegisterExtension]
7075
:RepetitionInfo: {javadoc-root}/org/junit/jupiter/api/RepetitionInfo.html[RepetitionInfo]
7176
:ResourceLock: {javadoc-root}/org/junit/jupiter/api/parallel/ResourceLock.html[@ResourceLock]
@@ -74,6 +79,7 @@ endif::[]
7479
:TestInfo: {javadoc-root}/org/junit/jupiter/api/TestInfo.html[TestInfo]
7580
:TestInstanceFactory: {javadoc-root}/org/junit/jupiter/api/extension/TestInstanceFactory.html[TestInstanceFactory]
7681
:TestInstancePostProcessor: {javadoc-root}/org/junit/jupiter/api/extension/TestInstancePostProcessor.html[TestInstancePostProcessor]
82+
:TestMethodOrder: {javadoc-root}/org/junit/jupiter/api/TestMethodOrder.html[@TestMethodOrder]
7783
:TestReporter: {javadoc-root}/org/junit/jupiter/api/TestReporter.html[TestReporter]
7884
:TestTemplate: {javadoc-root}/org/junit/jupiter/api/TestTemplate.html[@TestTemplate]
7985
:TestTemplateInvocationContext: {javadoc-root}/org/junit/jupiter/api/extension/TestTemplateInvocationContext.html[TestTemplateInvocationContext]

‎documentation/src/docs/asciidoc/release-notes/release-notes-5.4.0-M1.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,10 @@ repository on GitHub.
8383
* New `LOCALE` and `TIME_ZONE` constants in `org.junit.jupiter.api.parallel.Resources`
8484
for use with `@ResourceLock` to synchronize test execution regarding the default
8585
`Locale` and default `TimeZone`, respectively.
86+
* New `MethodOrderer` API for ordering the sequence of tests with built-in support for
87+
_alphanumeric_, `@Order` annotation based, and _random_ ordering of test methods.
88+
- See <<../user-guide/index.adoc#writing-tests-test-execution-order, Test Execution
89+
Order>> in the User Guide for details.
8690
* New `DisplayNameGenerator` interface and `@DisplayNameGeneration` annotation that allow
8791
declarative configuration of a pre-defined or custom display name generator.
8892
* JUnit 4's `@Ignore` annotation is now supported for disabling test classes and test

‎documentation/src/docs/asciidoc/user-guide/writing-tests.adoc

+39-1
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,9 @@ module.
2424
| `@ParameterizedTest` | Denotes that a method is a <<writing-tests-parameterized-tests, parameterized test>>. Such methods are _inherited_ unless they are _overridden_.
2525
| `@RepeatedTest` | Denotes that a method is a test template for a <<writing-tests-repeated-tests, repeated test>>. Such methods are _inherited_ unless they are _overridden_.
2626
| `@TestFactory` | Denotes that a method is a test factory for <<writing-tests-dynamic-tests, dynamic tests>>. Such methods are _inherited_ unless they are _overridden_.
27-
| `@TestInstance` | Used to configure the <<writing-tests-test-instance-lifecycle, test instance lifecycle>> for the annotated test class. Such annotations are _inherited_.
2827
| `@TestTemplate` | Denotes that a method is a <<writing-tests-test-templates, template for test cases>> designed to be invoked multiple times depending on the number of invocation contexts returned by the registered <<extensions-test-templates, providers>>. Such methods are _inherited_ unless they are _overridden_.
28+
| `@TestMethodOrder` | Used to configure the <<writing-tests-test-execution-order, test method execution order>> for the annotated test class; similar to JUnit 4's `@FixMethodOrder`. Such annotations are _inherited_.
29+
| `@TestInstance` | Used to configure the <<writing-tests-test-instance-lifecycle, test instance lifecycle>> for the annotated test class. Such annotations are _inherited_.
2930
| `@DisplayName` | Declares a custom display name for the test class or test method. Such annotations are not _inherited_.
3031
| `@DisplayNameGeneration` | Declares a custom display name generator for the test class. Such annotations are _inherited_.
3132
| `@BeforeEach` | Denotes that the annotated method should be executed _before_ *each* `@Test`, `@RepeatedTest`, `@ParameterizedTest`, or `@TestFactory` method in the current class; analogous to JUnit 4's `@Before`. Such methods are _inherited_ unless they are _overridden_.
@@ -363,6 +364,43 @@ characters have been removed.
363364
include::{testDir}/example/TaggingDemo.java[tags=user_guide]
364365
----
365366

367+
[[writing-tests-test-execution-order]]
368+
=== Test Execution Order
369+
370+
By default, test methods will be ordered using a default algorithm that is deterministic
371+
but intentionally nonobvious. This ensures that subsequent runs of a test suite execute
372+
test methods in the same order, thereby allowing for repeatable builds.
373+
374+
NOTE: In this context, a _test method_ is any instance method that is directly or
375+
meta-annotated with `@Test`, `@RepeatedTest`, `@ParameterizedTest`, `@TestFactory`, or
376+
`@TestTemplate`.
377+
378+
Although true _unit tests_ typically should not rely on the order in which they are
379+
executed, there are times when it is necessary to enforce a specific test method
380+
execution order -- for example, when writing _integration tests_ or _functional tests_
381+
where the sequence of the tests is important, especially in conjunction
382+
`@TestInstance(Lifecycle.PER_CLASS)`.
383+
384+
To control the order in which test methods are executed, annotate your test class or test
385+
interface with `{TestMethodOrder}` and specify the desired `{MethodOrderer}`
386+
implementation. You can implement your own custom `MethodOrderer` or use one of the
387+
following built-in `MethodOrderer` implementations.
388+
389+
* `{Alphanumeric}`: sorts test methods _alphanumerically_ based on their names and formal
390+
parameter lists.
391+
* `{OrderAnnotation}`: sorts test methods _numerically_ based on values specified via the
392+
`{Order}` annotation.
393+
* `{Random}`: orders test methods _pseudo-randomly_ and supports configuration of a
394+
custom _seed_.
395+
396+
The following example demonstrates how to guarantee that test methods are executed in the
397+
order specified via the `@Order` annotation.
398+
399+
[source,java,indent=0]
400+
----
401+
include::{testDir}/example/OrderedTestsDemo.java[tags=user_guide]
402+
----
403+
366404
[[writing-tests-test-instance-lifecycle]]
367405
=== Test Instance Lifecycle
368406

‎documentation/src/test/java/example/OrderedTestsDemo.java

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
* Copyright 2015-2018 the original author or authors.
3+
*
4+
* All rights reserved. This program and the accompanying materials are
5+
* made available under the terms of the Eclipse Public License v2.0 which
6+
* accompanies this distribution and is available at
7+
*
8+
* http://www.eclipse.org/legal/epl-v20.html
9+
*/
10+
11+
package example;
12+
13+
// tag::user_guide[]
14+
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
15+
import org.junit.jupiter.api.Order;
16+
import org.junit.jupiter.api.Test;
17+
import org.junit.jupiter.api.TestMethodOrder;
18+
19+
@TestMethodOrder(OrderAnnotation.class)
20+
class OrderedTestsDemo {
21+
22+
@Test
23+
@Order(1)
24+
void nullValues() {
25+
}
26+
27+
@Test
28+
@Order(2)
29+
void emptyValues() {
30+
}
31+
32+
@Test
33+
@Order(3)
34+
void validValues() {
35+
}
36+
}
37+
// end::user_guide[]

‎junit-jupiter-api/src/main/java/org/junit/jupiter/api/MethodDescriptor.java

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/*
2+
* Copyright 2015-2018 the original author or authors.
3+
*
4+
* All rights reserved. This program and the accompanying materials are
5+
* made available under the terms of the Eclipse Public License v2.0 which
6+
* accompanies this distribution and is available at
7+
*
8+
* http://www.eclipse.org/legal/epl-v20.html
9+
*/
10+
11+
package org.junit.jupiter.api;
12+
13+
import static org.apiguardian.api.API.Status.EXPERIMENTAL;
14+
15+
import java.lang.annotation.Annotation;
16+
import java.lang.reflect.Method;
17+
import java.util.List;
18+
import java.util.Optional;
19+
20+
import org.apiguardian.api.API;
21+
22+
/**
23+
* {@link MethodDescriptor} encapsulates functionality for a given {@link Method}.
24+
*
25+
* @since 5.4
26+
* @see MethodOrdererContext
27+
*/
28+
@API(status = EXPERIMENTAL, since = "5.4")
29+
public interface MethodDescriptor {
30+
31+
/**
32+
* Get the method for this descriptor.
33+
*
34+
* @return the method; never {@code null}
35+
*/
36+
Method getMethod();
37+
38+
/**
39+
* Determine if an annotation of {@code annotationType} is either
40+
* <em>present</em> or <em>meta-present</em> on the {@link Method} for
41+
* this descriptor.
42+
*
43+
* @param annotationType the annotation type to search for; never {@code null}
44+
* @return {@code true} if the annotation is present or meta-present
45+
* @see #findAnnotation(Class)
46+
* @see #findRepeatableAnnotations(Class)
47+
*/
48+
boolean isAnnotated(Class<? extends Annotation> annotationType);
49+
50+
/**
51+
* Find the first annotation of {@code annotationType} that is either
52+
* <em>present</em> or <em>meta-present</em> on the {@link Method} for
53+
* this descriptor.
54+
*
55+
* @param <A> the annotation type
56+
* @param annotationType the annotation type to search for; never {@code null}
57+
* @return an {@code Optional} containing the annotation; never {@code null} but
58+
* potentially empty
59+
* @see #isAnnotated(Class)
60+
* @see #findRepeatableAnnotations(Class)
61+
*/
62+
<A extends Annotation> Optional<A> findAnnotation(Class<A> annotationType);
63+
64+
/**
65+
* Find all <em>repeatable</em> {@linkplain Annotation annotations} of
66+
* {@code annotationType} that are either <em>present</em> or
67+
* <em>meta-present</em> on the {@link Method} for this descriptor.
68+
*
69+
* @param <A> the annotation type
70+
* @param annotationType the repeatable annotation type to search for; never
71+
* {@code null}
72+
* @return the list of all such annotations found; neither {@code null} nor
73+
* mutable, but potentially empty
74+
* @see #isAnnotated(Class)
75+
* @see #findAnnotation(Class)
76+
* @see java.lang.annotation.Repeatable
77+
*/
78+
<A extends Annotation> List<A> findRepeatableAnnotations(Class<A> annotationType);
79+
80+
}

0 commit comments

Comments
 (0)