Skip to content

Commit 75dd9b6

Browse files
stmontgomerystmontgomery
committed
Introduce custom test execution trait API
1 parent 05890bc commit 75dd9b6

File tree

5 files changed

+466
-51
lines changed

5 files changed

+466
-51
lines changed

Diff for: ‎Documentation/Proposals/NNNN-custom-test-execution-traits.md

+376
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
# Custom Test Execution Traits
2+
3+
* Proposal: [SWT-NNNN](NNNN-filename.md)
4+
* Authors: [Stuart Montgomery](https://github.com/stmontgomery)
5+
* Status: **Awaiting review**
6+
* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86)
7+
* Review: ([pitch](https://forums.swift.org/...))
8+
9+
## Introduction
10+
11+
This introduces API which enables a custom `Trait`-conforming type to customize
12+
the execution of test functions and suites, including running code before or
13+
after them.
14+
15+
## Motivation
16+
17+
One of the primary motivations for the trait system in Swift Testing, as
18+
[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility),
19+
is to provide a way to customize the behavior of tests which have things in
20+
common. If all the tests in a given suite type need the same custom behavior,
21+
`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of
22+
the tests in a suite need custom behavior, or tests across different levels of
23+
the suite hierarchy need it, traits would be a good place to encapsulate common
24+
logic since they can be applied granularly per-test or per-suite. This aspect of
25+
the vision for traits hasn't been realized yet, though: the `Trait` protocol
26+
does not offer a way for a trait to customize the execution of the tests or
27+
suites it's applied to.
28+
29+
Customizing a test's behavior typically means running code either before or
30+
after it runs, or both. Consolidating common set-up and tear-down logic allows
31+
each test function to be more succinct with less repetitive boilerplate so it
32+
can focus on what makes it unique.
33+
34+
## Proposed solution
35+
36+
At a high level, this proposal entails adding API to the `Trait` protocol
37+
allowing a conforming type to opt-in to customizing the execution of test
38+
behavior. We discuss how that capability should be exposed to trait types below.
39+
40+
### Supporting scoped access
41+
42+
There are different approaches one could take to expose hooks for a trait to
43+
customize test behavior. To illustrate one of them, consider the following
44+
example of a `@Test` function with a custom trait whose purpose is to set mock
45+
API credentials for the duration of each test it's applied to:
46+
47+
```swift
48+
@Test(.mockAPICredentials)
49+
func example() {
50+
// ...
51+
}
52+
53+
struct MockAPICredentialsTrait: TestTrait { ... }
54+
55+
extension Trait where Self == MockAPICredentialsTrait {
56+
static var mockAPICredentials: Self { ... }
57+
}
58+
```
59+
60+
In this hypothetical example, the current API credentials are stored via a
61+
static property on an `APICredentials` type which is part of the module being
62+
tested:
63+
64+
```swift
65+
struct APICredentials {
66+
var apiKey: String
67+
68+
static var shared: Self?
69+
}
70+
```
71+
72+
One way that this custom trait could customize the API credentials during each
73+
test is if the `Trait` protocol were to expose a pair of method requirements
74+
which were then called before and after the test, respectively:
75+
76+
```swift
77+
public protocol Trait: Sendable {
78+
// ...
79+
func setUp() async throws
80+
func tearDown() async throws
81+
}
82+
83+
extension Trait {
84+
// ...
85+
public func setUp() async throws { /* No-op */ }
86+
public func tearDown() async throws { /* No-op */ }
87+
}
88+
```
89+
90+
The custom trait type could adopt these using code such as the following:
91+
92+
```swift
93+
extension MockAPICredentialsTrait {
94+
func setUp() {
95+
APICredentials.shared = .init(apiKey: "...")
96+
}
97+
98+
func tearDown() {
99+
APICredentials.shared = nil
100+
}
101+
}
102+
```
103+
104+
Many testing systems use this pattern, including XCTest. However, this approach
105+
encourages the use of global mutable state such as the `APICredentials.shared`
106+
variable, and this limits the testing library's ability to parallelize test
107+
execution, which is
108+
[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency).
109+
110+
The use of nonisolated static variables is generally discouraged now, and in
111+
Swift 6 the above `APICredentials.shared` property produces an error. One way
112+
to resolve that is to change it to a `@TaskLocal` variable, as this would be
113+
concurrency-safe and still allow tests accessing this state to run in parallel:
114+
115+
```swift
116+
extension APICredentials {
117+
@TaskLocal static var current: Self?
118+
}
119+
```
120+
121+
Binding task local values requires using the scoped access
122+
[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)
123+
API though, and that would not be possible if `Trait` exposed separate methods
124+
like `setUp()` and `tearDown()`.
125+
126+
For these reasons, I believe it's important to expose this trait capability
127+
using a single, scoped access-style API which accepts a closure. A simplified
128+
version of that idea might look like this:
129+
130+
```swift
131+
public protocol Trait: Sendable {
132+
// ...
133+
134+
// Simplified example, not the actual proposal
135+
func executeTest(_ body: @Sendable () async throws -> Void) async throws
136+
}
137+
138+
extension MockAPICredentialsTrait {
139+
func executeTest(_ body: @Sendable () async throws -> Void) async throws {
140+
let mockCredentials = APICredentials(apiKey: "...")
141+
try await APICredentials.$current.withValue(mockCredentials) {
142+
try await body()
143+
}
144+
}
145+
}
146+
```
147+
148+
### Avoiding unnecessarily lengthy backtraces
149+
150+
A scoped access-style API has some potential downsides. To apply this approach
151+
to a test function, the scoped call of a trait must wrap the invocation of that
152+
test function, and every _other_ trait applied to that same test which offers
153+
custom behavior _also_ must wrap the other traits' calls in a nesting fashion.
154+
To visualize this, imagine a test function with multiple traits:
155+
156+
```swift
157+
@Test(.traitA, .traitB, .traitC)
158+
func exampleTest() {
159+
// ...
160+
}
161+
```
162+
163+
If all three of those traits customize test execution behavior, then each of
164+
them needs to wrap the call to the next one, and the last trait needs to wrap
165+
the invocation of the test, illustrated by the following:
166+
167+
```
168+
TraitA.executeTest {
169+
TraitB.executeTest {
170+
TraitC.executeTest {
171+
exampleTest()
172+
}
173+
}
174+
}
175+
```
176+
177+
Tests may have an arbitrary number of traits applied to them, including those
178+
inherited from containing suite types. A naïve implementation in which _every_
179+
trait is given the opportunity to customize test behavior by calling its scoped
180+
access API might cause unnecessarily lengthy backtraces that make debugging the
181+
body of tests more difficult. Or worse: if the number of traits is great enough,
182+
it could cause a stack overflow.
183+
184+
In practice, most traits probably will _not_ need to customize test behavior, so
185+
to mitigate these downsides it's important that there be some way to distinguish
186+
traits which customize test behavior. That way, the testing library can limit
187+
these scoped access calls to only the traits which require it.
188+
189+
## Detailed design
190+
191+
I propose the following new APIs:
192+
193+
- A new protocol `CustomTestExecuting` with a single required `execute(...)`
194+
method. This will be called to run a test, and allows the conforming type to
195+
perform custom logic before or after.
196+
- A new property `customTestExecutor` on the `Trait` protocol whose type is an
197+
`Optional` value of a type conforming to `CustomTestExecuting`. A `nil` value
198+
from this property will skip calling the `execute(...)` method.
199+
- A default implementation of `Trait.customTestExecutor` whose value is `nil`.
200+
- A conditional implementation of `Trait.customTestExecutor` whose value is
201+
`self` in the common case where the trait type conforms to
202+
`CustomTestExecuting` itself.
203+
204+
Since the `customTestExecutor` property is optional and `nil` by default, the
205+
testing library cannot invoke the `execute(...)` method unless a trait
206+
customizes test behavior. This avoids the "unnecessarily lengthy backtraces"
207+
problem above.
208+
209+
Below are the proposed interfaces:
210+
211+
```swift
212+
/// A protocol that allows customizing the execution of a test function (and
213+
/// each of its cases) or a test suite by performing custom code before or after
214+
/// it runs.
215+
public protocol CustomTestExecuting: Sendable {
216+
/// Execute a function for the specified test and/or test case.
217+
///
218+
/// - Parameters:
219+
/// - function: The function to perform. If `test` represents a test suite,
220+
/// this function encapsulates running all the tests in that suite. If
221+
/// `test` represents a test function, this function is the body of that
222+
/// test function (including all cases if it is parameterized.)
223+
/// - test: The test under which `function` is being performed.
224+
/// - testCase: The test case, if any, under which `function` is being
225+
/// performed. This is `nil` when invoked on a suite.
226+
///
227+
/// - Throws: Whatever is thrown by `function`, or an error preventing
228+
/// execution from running correctly.
229+
///
230+
/// This function is called for each ``Trait`` on a test suite or test
231+
/// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``.
232+
/// It allows additional work to be performed before or after the test runs.
233+
///
234+
/// This function is invoked once for the test its associated trait is applied
235+
/// to, and then once for each test case in that test, if applicable. If a
236+
/// test is skipped, this function is not invoked for that test or its cases.
237+
///
238+
/// Issues recorded by this function are associated with `test`.
239+
func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws
240+
}
241+
242+
public protocol Trait: Sendable {
243+
// ...
244+
245+
/// The type of the custom test executor for this trait.
246+
///
247+
/// The default type is `Never`.
248+
associatedtype CustomTestExecutor: CustomTestExecuting = Never
249+
250+
/// The custom test executor for this trait, if any.
251+
///
252+
/// If this trait's type conforms to ``CustomTestExecuting``, the default
253+
/// value of this property is `self` and this trait will be used to customize
254+
/// test execution. This is the most straightforward way to implement a trait
255+
/// which customizes the execution of tests.
256+
///
257+
/// However, if the value of this property is an instance of another type
258+
/// conforming to ``CustomTestExecuting``, that instance will be used to
259+
/// perform custom test execution instead. Otherwise, the default value of
260+
/// this property is `nil` (with the default type `Never?`), meaning that
261+
/// custom test execution will not be performed for tests this trait is
262+
/// applied to.
263+
var customTestExecutor: CustomTestExecutor? { get }
264+
}
265+
266+
extension Trait {
267+
// ...
268+
269+
// The default implementation, which returns `nil`.
270+
public var customTestExecutor: CustomTestExecutor? { get }
271+
}
272+
273+
extension Trait where CustomTestExecutor == Self {
274+
// Returns `self`.
275+
public var customTestExecutor: CustomTestExecutor? { get }
276+
}
277+
278+
extension Never: CustomTestExecuting {
279+
public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws
280+
}
281+
```
282+
283+
Here is a complete example of the usage scenario described earlier, showcasing
284+
the proposed APIs:
285+
286+
```swift
287+
@Test(.mockAPICredentials)
288+
func example() {
289+
// ...validate API usage, referencing `APICredentials.current`...
290+
}
291+
292+
struct MockAPICredentialsTrait: TestTrait, CustomTestExecuting {
293+
func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws {
294+
let mockCredentials = APICredentials(apiKey: "...")
295+
try await APICredentials.$current.withValue(mockCredentials) {
296+
try await function()
297+
}
298+
}
299+
}
300+
301+
extension Trait where Self == MockAPICredentialsTrait {
302+
static var mockAPICredentials: Self {
303+
Self()
304+
}
305+
}
306+
```
307+
308+
## Source compatibility
309+
310+
The proposed APIs are purely additive.
311+
312+
## Integration with supporting tools
313+
314+
Although some built-in traits are relevant to supporting tools (such as
315+
SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are
316+
only relevant within the test executable process while tests are running. We
317+
don't anticipate any particular need for this feature to integrate with
318+
supporting tools.
319+
320+
## Future directions
321+
322+
Some test authors have expressed interest in allowing custom traits to access
323+
the instance of a suite type for `@Test` instance methods, so the trait could
324+
inspect or mutate the instance. Currently, only instance-level members of a
325+
suite type (including `init`, `deinit`, and the test function itself) can access
326+
`self`, so this would grant traits applied to an instance test method access to
327+
the instance as well. This is certainly interesting, but poses several technical
328+
challenges that puts it out of scope of this proposal.
329+
330+
## Alternatives considered
331+
332+
### Separate set up & tear down methods on `Trait`
333+
334+
This idea was discussed in [Supporting scoped access](#supporting-scoped-access)
335+
above, and as mentioned there, the primary problem with this approach is that it
336+
cannot be used with scoped access-style APIs, including (importantly)
337+
`TaskLocal.withValue()`. For that reason, it prevents using that common Swift
338+
concurrency technique and reduces the potential for test parallelization.
339+
340+
### Add `execute(...)` directly to the `Trait` protocol
341+
342+
The proposed `execute(...)` method could be added as a requirement of the
343+
`Trait` protocol instead of being part of a separate `CustomTestExecuting`
344+
protocol, and it could have a default implementation which directly invokes the
345+
passed-in closure. But this approach would suffer from the lengthy backtrace
346+
problem described above.
347+
348+
### Extend the `Trait` protocol
349+
350+
The original SPI implementation of this feature included a protocol named
351+
`CustomExecutionTrait` which extended `Trait` and had roughly the same method
352+
requirement as the `CustomTestExecuting` protocol proposed above. This design
353+
worked, provided scoped access, and avoided the lengthy backtrace problem.
354+
355+
After evaluating the design and usage of this SPI though, it seemed unfortunate
356+
to structure it as a sub-protocol of `Trait` because it means that the full
357+
capabilities of the trait system are spread across multiple protocols. In the
358+
proposed design, the ability to provide a custom test executor value is exposed
359+
via the main `Trait` protocol, and it relies on an associated type to
360+
conditionally opt-in to custom test behavior. In other words, the proposed
361+
design expresses custom test behavior as just a _capability_ that a trait may
362+
have, rather than a distinct sub-type of trait.
363+
364+
Also, the implementation of this approach within the testing library was not
365+
ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at
366+
runtime, in contrast to the simpler and more performant Optional property of the
367+
proposed API.
368+
369+
## Acknowledgments
370+
371+
Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally
372+
implementing this as SPI, and for helping promote its usefulness.
373+
374+
Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas
375+
to refine the API, and considering alternatives to avoid unnecessarily long
376+
backtraces.

0 commit comments

Comments
 (0)