[SWT-NNNN] Introduce API allowing traits to customize test execution #733

stmontgomery · 2024-09-25T21:12:27Z

This includes an API proposal and code changes to introduce new API for custom traits to customize test execution.

View the API proposal for more details.

Motivation:

One of the primary motivations for the trait system in Swift Testing, as described in the vision document, is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, init and/or deinit (if applicable) can be used today. But if only some of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the Trait protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to.

Customizing a test's behavior typically means running code either before or after it runs, or both. Consolidating common set-up and tear-down logic allows each test function to be more succinct with less repetitive boilerplate so it can focus on what makes it unique.

Checklist:

Code and documentation should follow the style of the Style Guide.
If public symbols are renamed or modified, DocC references should be updated.

stmontgomery · 2024-09-25T21:21:05Z

@swift-ci please test Windows

Documentation/Proposals/NNNN-custom-test-execution-traits.md

grynspan · 2024-09-25T21:28:17Z

Documentation/Proposals/NNNN-custom-test-execution-traits.md

+struct MockAPICredentialsTrait: TestTrait, CustomTestExecuting {
+  func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws {
+    let mockCredentials = APICredentials(apiKey: "...")
+    try await APICredentials.$current.withValue(mockCredentials) {


We think task locals are going to be a particularly common use case, right? Perhaps we should build out a trait type that sets a task local:

@Test(.withValue(123, for: $foo)) func f() { ... }

Or even if we don't make it a trait, can we make it easier to write a trait that sets a task local with minimal boilerplate?

Agree that would be a very useful, built-in trait to have. Let's consider that separately, since it would ultimately leverage the API proposed here

Doesn't need to be part of this proposal, but I think it does affect this proposal. Might be worth coming up with an example of when you'd use this stuff other than to set a task local.

Sure. Briefly: not all work before/after a test is simply mutating state in that process. It may involve writing to the filesystem, or interacting with other system-wide resources. (Clearly, parallelism would need to be considered in such scenarios.)

Also: Users can still use this facility to mutate global or static state, as long as it complies with Swift concurrency rules

Can we make sure to add this example to the discussion?

Where, specifically, do you recommend we place this example code snippet?

Documentation/Proposals/NNNN-custom-test-execution-traits.md

Sources/Testing/Testing.docc/Traits/Trait.md

Sources/Testing/Traits/Trait.swift

…e per scope, by default

stmontgomery · 2024-11-20T19:48:46Z

@stmontgomery Per our conversation offline you mentioned that order of execution of custom execution traits is well-defined to be outer-to-inner, left-to-right -- it would be great if we could explicitly add a note about that in the documentation so that folks are aware of the expected behavior!

I've added language specifying this ordering to the updated proposal and PR

… the base verb instead of "execute"

stmontgomery · 2024-11-20T20:38:41Z

@swift-ci please test

Documentation/Proposals/NNNN-custom-test-execution-traits.md

Sources/Testing/Traits/Trait.swift

Documentation/Proposals/NNNN-custom-test-execution-traits.md

…ntation

stmontgomery · 2025-01-07T20:55:30Z

@swift-ci please test

Assign the "Test Scoping Traits" proposal, which was recently approved and merged in #733, a number ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

This is a follow-on from #733 which introduced the Test Scoping Traits feature. As that proposal indicated, the `SPIAwareTrait` SPI protocol is no longer necessary, so this PR removes it and adopts `TestScoping` in its place. ### Motivation: Adopting Test Scoping Traits for the purpose which `SPIAwareTrait` previously served simplifies a lot of runner and planning logic. ### Modifications: - Delete `SPIAwareTrait` and related SPI. - Adopt `TestScoping` in `ParallelizationTrait` instead. - Changed `ParallelizationTrait` to _not_ always have the value `true` for `isRecursive`. Test Scoping Traits makes this no longer necessary: the invocation of children of any suites which have `.serialized` is scoped to have parallelization disabled. - Removed tracking of `isParallelizationEnabled` on the `Runner.Plan.Action.RunOptions` type. For now, I kept the `RunOptions` struct, since it could be useful in the future and it's SPI. - Removed obsolete logic in `Runner` for propagating ancestor steps, and refactored slightly to make many things `static`. This is to help ensure certain functions don't access the `self.configuration` property when instead they should be accessing `Configuration.current`. ### Result: No functional change, but implementation is simplified. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

…wiftlang#733) This includes an API proposal and code changes to introduce new API for custom traits to customize test execution. View the [API proposal](https://github.com/stmontgomery/swift-testing/blob/publicize-CustomExecutionTrait/Documentation/Proposals/NNNN-custom-test-execution-traits.md) for more details. One of the primary motivations for the trait system in Swift Testing, as [described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, `init` and/or `deinit` (if applicable) can be used today. But if only _some_ of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the `Trait` protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to. Customizing a test's behavior typically means running code either before or after it runs, or both. Consolidating common set-up and tear-down logic allows each test function to be more succinct with less repetitive boilerplate so it can focus on what makes it unique. - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

@grynspan

…ution (#907) - **Explanation**: Include the Test Scoping Traits feature and proposal in Swift 6.1. - **Scope**: Adds an important and frequently requested new feature to the 6.1 release. Removes an SPI which may be in use, but was always intended to be refined/replaced. - **Issues**: n/a - **Original PRs**: #733, #900 - **Risk**: Low, new functionality - **Testing**: New APIs have unit tests. - **Reviewers**: @grynspan, @briancroom

stmontgomery · 2025-03-05T13:18:46Z

Resolves #36

stmontgomery added documentation 📚 Improvements or additions to documentation enhancement New feature or request public-api Affects public API api-proposal API proposal PRs (documentation only) labels Sep 25, 2024

stmontgomery added this to the Swift 6.1 milestone Sep 25, 2024

stmontgomery self-assigned this Sep 25, 2024

stmontgomery requested review from grynspan, dennisweissmann, briancroom, SeanROlszewski and suzannaratcliff as code owners September 25, 2024 21:12

stmontgomery force-pushed the publicize-CustomExecutionTrait branch from 75975cf to 75dd9b6 Compare September 25, 2024 21:18

grynspan changed the title ~~Introduce API allowing traits to customize test execution~~ [SWT-NNNN] Introduce API allowing traits to customize test execution Sep 25, 2024