Include the location of a parameterized test argument which recorded an issue (if possible) in console output for better traceability

[ojun9]

Motivation

When using @Test(arguments:) to test multiple input patterns in a single test function, it can be difficult to identify which argument caused a failure, especially when several similar test cases are used.

For example, consider the following test setup:

Supporting code:

struct User: Equatable {
    let name: String
    let age: Int
    let email: String
    let username: String
    let password: String
    let isActive: Bool
    let signupDate: Date
    let lastLoginDate: Date
    let phoneNumber: String
    let address: String
    let bio: String
}

extension User {
    static func mock(
        name: String = "John Doe",
        age: Int = 30,
        email: String = "[email protected]",
        username: String = "johndoe",
        password: String = "password123",
        isActive: Bool = true,
        signupDate: Date = Date(),
        lastLoginDate: Date = Date(),
        phoneNumber: String = "000-0000-0000",
        address: String = "123 Apple Street",
        bio: String = "Just a test user."
    ) -> Self {
        .init(
            name: name,
            age: age,
            email: email,
            username: username,
            password: password,
            isActive: isActive,
            signupDate: signupDate,
            lastLoginDate: lastLoginDate,
            phoneNumber: phoneNumber,
            address: address,
            bio: bio
        )
    }
}

Test code:

struct UserTests {
    @Test(arguments: [
        User.mock(),
        .mock(name: "hoge", isActive: false),
        .mock(lastLoginDate: .distantPast, bio: "Hello World"),

        // Multiple mock cases follow

        .mock(email: "[email protected]"),
        .mock(phoneNumber: "999-9999-9999")
    ])
    func someTest(_ user: User) async throws {
        let expected: User = .mock() // In practice, this would call some logic
        #expect(user == expected)
    }
}

When a test fails, the output looks like this:

✘ Test someTest(_:) recorded an issue with 1 argument user → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World") at TestTests.swift:72:9: Expectation failed: (user → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World")) == (expected → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 2025-03-25 09:59:04 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Just a test user."))

However, this output alone doesn’t clearly indicate which argument (i.e., which entry in the arguments array) caused the failure. This makes it difficult to trace which case needs to be fixed, especially in large or complex test sets.

Proposed solution

A helpful enhancement would be to include the argument’s index in the failure output. For example, if the second item in the arguments array fails, the output might look like this:

✘ Test someTest(_:) [arguments index: 2] recorded an issue with 1 argument user → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World") at TestTests.swift:72:9: Expectation failed: (user → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World")) == (expected → User(name: "John Doe", age: 30, email: "[email protected]", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 2025-03-25 09:59:04 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Just a test user."))

This small addition would make it significantly easier to identify the failing case and improve the debugging experience, especially when dealing with multiple test arguments.

Currently, developers can work around this by wrapping the argument in a custom struct with a label, like so:

struct UserTestCase {
    let label: String
    let user: User
}

@Test(arguments: [
    UserTestCase(label: "default", user: .mock()),
    UserTestCase(label: "inactive", user: .mock(name: "hoge", isActive: false)),
])
func someTest(_ testCase: UserTestCase) async throws {
    #expect(testCase.user == .mock(), "Failure in case: \(testCase.label)")
}

While this pattern works, it adds boilerplate and isn’t ideal for simpler test cases.
Including the argument index in the default output would be a lightweight and widely useful improvement that benefits many test scenarios.

Alternatives considered

No response

Additional information

No response

[grynspan]

We intentionally don't include an integer index because we cannot reliably determine them at runtime for all collection types, and cannot reliably determine if a given collection type has stable indexing.

Improvements to test case enumeration are currently tracked by #671.

[stmontgomery]

Improvements to test case enumeration are currently tracked by #671.

And I'm pleased to report that one subtask of that, #1000, recently landed and provides a mechanism to distinguish test case IDs.

That being said, I think neither #671 nor #1000 will truly address the complaint here. It seems to me that the basic problem is that when a test case for a parameterized test argument records an issue, it can be difficult to trace back to the location in source where that argument was defined. #1000 can help address that issue, but won't solve it on its own.

I think we should find a way to capture more information about the source and location of test arguments so that in the results we could actually note the source location where the argument was passed. That, to me, would be even more powerful than including each arguments' index in the collection; an index is just an indirect way to refer to an element, but if you can jump directly to the location of that element in source code, all the better. (This all assumes that a collection literal is being passed to @Test(arguments:), of course; fully dynamic construction of test arguments could not benefit from this idea since the macro isn't exposed to them.) I'll hold this issue to track this larger idea, since it has come up before.

[stmontgomery]

Ah, here we go: this can actually be considered a dupe of a different issue: #519.

[ojun9]

Thanks for the detailed explanation!
That makes a lot of sense — being able to trace the exact source location of a test argument would be incredibly helpful. I'm glad this idea is being tracked, and I’d be happy to help if there’s any way to contribute to it.

[grynspan]

That's a better original, thanks :)