Allow direct creation of Issues

[younata]

Description

Xcode's XCTest has the XCTIssue API for reporting issues, but the closest thing to creating and reporting an XCTIssue is Issue.record. Which has limitations and prevents me from providing semantics for why an expectation failed.

This limits the ability of Nimble to integrate with Swift Testing.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Discussed in a lab.

We've so far resisted doing this in the general case because we want to steer developers toward specific API like #expect(). @younata's use case is integration with tools however, which requires a different set of interfaces from what we provide for test authors.

We currently offer a number of interfaces guarded by @_spi(ForToolsIntegrationOnly). These interfaces are available when building Swift Testing from a package. However, SPI is not available when using the copy of Swift Testing included in Xcode 16.

Would it be useful if we added an Issue.init member guarded by @_spi in this way? Would that be sufficient for your needs?

[younata]

Hi @grynspan! Thanks for the quick response!

My concern with using @_spi (which I am unfamiliar with) is if this will affect Nimble's Cocoapods & Carthage integrations. For the Swift Packages integration, @_spi will work fine. But, naively, I'm planning to try to get Nimble's Cocoapods & Carthage integrations to use the built-in Swift Testing. Last I checked (and google isn't helping me here).
This is because there isn't a way for me to tell Cocoapods that I have a dependency on something that's only available as a swift package. I know for a fact that if I were to embed Swift Testing as a Package for the Carthage integration, then the tests will refuse to build unless you also add the Swift Testing package as a dependency to your app (which I know because Nimble's Carthage integration is currently broken for the same reason).

Blocking the integration on tools-only @_spi looks promising, but I have reservations about if it will work with the other package managers Nimble purports to support.

[stmontgomery]

I think we do need to expose the ability to programmatically initialize and record an Issue and have that eventually become regular API. I can take this. I can't currently commit to any particular timeframe for delivering this functionality, however.

If anyone would like to get started drafting an API proposal (using our template) for this, that would certainly be welcome!

[stmontgomery]

Tracked internally as rdar://114106842

[younata]

I can't currently commit to any particular timeframe for delivering this functionality, however.

My only timeline is that I'd like to be able to ship Swift Testing integration in Nimble when Xcode 16 is fully released.

If anyone would like to get started drafting an API proposal (using our template) for this, that would certainly be welcome!

I'm happy to take on that API proposal. It'll probably go through a few revisions before it's merged, though.

[younata]

Let's go with #490, as described in #481 (comment)

[mbrandonw]

We currently offer a number of interfaces guarded by @_spi(ForToolsIntegrationOnly). These interfaces are available when building Swift Testing from a package. However, SPI is not available when using the copy of Swift Testing included in Xcode 16.

Hi @grynspan, sorry to comment on a closed issue, but it seems that @_spi import is working just fine for the Testing framework in Xcode. Is this officially supported to get access to the integration tools?

[grynspan]

@mbrandonw No, it's a bug in Xcode. Apple is investigating.

[mbrandonw]

Hi @grynspan, thanks! And good to know. Is there any update on the status of CustomExecutionTrait, and whether or not it will make it to the first official Xcode release of swift-testing?

[grynspan]

This is not the appropriate place for questions about Swift Testing in general. Please consider starting a thread on the Swift forums. Thanks!