Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define and implement a stable JSON schema for output. #368

Closed
plemarquand opened this issue Apr 22, 2024 · 3 comments · Fixed by #383
Closed

Define and implement a stable JSON schema for output. #368

plemarquand opened this issue Apr 22, 2024 · 3 comments · Fixed by #383
Assignees
Labels
enhancement New feature or request tools integration Integration of swift-testing into tools/IDEs

Comments

@plemarquand
Copy link
Contributor

Description

Experimental event streaming support was added in #286. Issues are reported as individual events with richly structured JSON that describe the failure. This is great for tools that need to work with and transform raw issues, but many tools will just want to use the human readable failure string, i.e: Expectation failed: 1 == 2.

Swift-testing has done a lot of work to build up a nice message, and it would be nice to avoid reimplementing this in other tools (and keeping it up to date).

Feature: Add a new message field to the issueRecorded event that provides the human readable failure string. Bonus points if there is a markdown version of the string, as many tools can render markdown directly.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

@plemarquand plemarquand added the enhancement New feature or request label Apr 22, 2024
@plemarquand
Copy link
Contributor Author

As an example @grynspan suggested something like:

 {
  "eventKind": "issueRecorded",
  "message": {
    "symbol": "failure",
    "string": "Issue exploded! Oh no!",
    "markdown": "Issue **exploded**! [Oh no!](ohno.gif)"
  },
  "sourceLocation": { fileID: "Module/File.swift", line: 123, column: 456 },
  "instant": { "seconds": 12345, "attoseconds": 123450000000000 },
  ...
}

@grynspan
Copy link
Contributor

As of right now, the experimental event stream just encodes the fields of the events themselves. In order to make a stable API/ABI, we likely need to change that to a formal JSON schema with dedicated and intentional content, not just "whatever JSONEncoder produced."

@grynspan grynspan added the tools integration Integration of swift-testing into tools/IDEs label Apr 22, 2024
@grynspan grynspan changed the title Add a message field to the issueRecorded event in the experimental JSON event stream Define and implement a stable JSON schema for output. May 1, 2024
@grynspan
Copy link
Contributor

grynspan commented May 1, 2024

Tracked internally as rdar://126858814.

@grynspan grynspan self-assigned this May 1, 2024
grynspan added a commit that referenced this issue May 10, 2024
This PR defines a JSON schema for events output from swift-testing via
either `--experimental-event-stream-output` or `abiEntryPoint_v0()`. The
JSON schema needs to be formally reviewed separately.

We are already writing an experimental JSON stream using "snapshot"
types and we don't want to break folks experimenting with it before we
have a final JSON schema, so for now that remains the default. To opt
into the new schema, pass `--experimental-event-stream-version 0` (note
that this argument is not forwarded from `swift test`. See
swiftlang/swift-package-manager#7534.)

Resolves #368.

### Checklist:

- [x] Code and documentation should follow the style of the [Style
Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md).
- [x] If public symbols are renamed or modified, DocC references should
be updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request tools integration Integration of swift-testing into tools/IDEs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants