Skip to content

Commit d5f9851

Browse files
grynspangrynspan
authored
Define and implement a strawman stable JSON schema for output. (#383)
This PR defines a JSON schema for events output from swift-testing via either `--experimental-event-stream-output` or `abiEntryPoint_v0()`. The JSON schema needs to be formally reviewed separately. We are already writing an experimental JSON stream using "snapshot" types and we don't want to break folks experimenting with it before we have a final JSON schema, so for now that remains the default. To opt into the new schema, pass `--experimental-event-stream-version 0` (note that this argument is not forwarded from `swift test`. See swiftlang/swift-package-manager#7534.) Resolves #368. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.
1 parent a1301c6 commit d5f9851

19 files changed

+936
-125
lines changed

‎Documentation/ABI/JSON.md

+220
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# JSON schema
2+
3+
<!--
4+
This source file is part of the Swift.org open source project
5+
6+
Copyright (c) 2024 Apple Inc. and the Swift project authors
7+
Licensed under Apache License v2.0 with Runtime Library Exception
8+
9+
See https://swift.org/LICENSE.txt for license information
10+
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
11+
-->
12+
13+
This document outlines the JSON schemas used by the testing library for its ABI
14+
entry point and for the `--experimental-event-stream-output` command-line
15+
argument. For more information about the ABI entry point, see the documentation
16+
for [ABI.EntryPoint_v0](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint_v0&type=code).
17+
18+
> [!WARNING]
19+
> This JSON schema is still being developed and is subject to any and all
20+
> changes including removal from the package.
21+
22+
## Modified Backus-Naur form
23+
24+
This schema is expressed using a modified Backus-Naur syntax. `{`, `}`, `:`, and
25+
`,` represent their corresponding JSON tokens. `\n` represents an ASCII newline
26+
character.
27+
28+
The order of keys in JSON objects is not normative. Whitespace in this schema is
29+
not normative; it is present to help the reader understand the content of the
30+
various JSON objects in the schema. The event stream is output using the JSON
31+
Lines format and does not include newline characters (except **one** at the end
32+
of the `<output-record-line>` rule.)
33+
34+
Trailing commas in JSON objects and arrays are only to be included where
35+
syntactically valid.
36+
37+
### Common data types
38+
39+
`<string>` and `<number>` are defined as in JSON. `<array:T>` represents an
40+
array (also defined as in JSON) whose elements all follow rule `<T>`.
41+
42+
```
43+
<bool> ::= true | false ; as in JSON
44+
45+
<source-location> ::= {
46+
"fileID": <string>, ; the Swift file ID of the file
47+
"line": <number>,
48+
"column": <number>,
49+
}
50+
51+
<instant> ::= {
52+
"absolute": <number>, ; floating-point seconds since system-defined epoch
53+
"since1970": <number>, ; floating-point seconds since 1970-01-01 00:00:00 UT
54+
}
55+
56+
<version> ::= "version": 0 ; will be incremented as the format changes
57+
```
58+
59+
<!--
60+
TODO: implement input/configuration
61+
62+
### Configuration
63+
64+
A single configuration is passed into the testing library prior to running any
65+
tests and, as the name suggests, configures the test run. The configuration is
66+
encoded as a single [JSON Lines](https://jsonlines.org) value.
67+
68+
```
69+
<configuration-record> ::= {
70+
<version>,
71+
"kind": "configuration",
72+
"payload": <configuration>
73+
}
74+
75+
<configuration> ::= {
76+
["verbosity": <number>,] ; 0 is the default; higher means more verbose output
77+
; while negative values mean quieter output.
78+
["filters": <array:test-filter>,] ; how to filter the tests in the test run
79+
["parallel": <bool>,] ; whether to enable parallel testing (on by default)
80+
; more TBD
81+
}
82+
83+
<test-filter> ::= <test-filter-tag> | <test-filter-id>
84+
85+
<test-filter-action> ::= "include" | "exclude"
86+
87+
<test-filter-tag> ::= {
88+
"action": <test-filter-action>,
89+
"tags": <array:string>, ; the names of tags to include
90+
"operator": <test-filter-tag-operator> ; how to combine the values in "tags"
91+
}
92+
93+
<test-filter-tag-operator> ::= "any" | "all"
94+
95+
<test-filter-id> ::= {
96+
"action": <test-filter-action>,
97+
"id": <test-id> ; the ID of the test to filter in/out
98+
}
99+
```
100+
-->
101+
102+
### Streams
103+
104+
A stream consists of a sequence of values encoded as [JSON Lines](https://jsonlines.org).
105+
A single instance of `<output-stream>` is defined per test process and can be
106+
accessed by passing `--experimental-event-stream-output` to the test executable
107+
created by `swift build --build-tests`.
108+
109+
```
110+
<output-stream> ::= <output-record>\n | <output-record>\n <output-stream>
111+
```
112+
113+
### Records
114+
115+
Records represent the values produced on a stream. Each record is encoded on a
116+
single line and can be decoded independently of other lines. If a decoder
117+
encounters a record whose `"kind"` field is unrecognized, the decoder should
118+
ignore that line.
119+
120+
```
121+
<output-record> ::= <test-record> | <event-record>
122+
123+
<test-record> ::= {
124+
<version>,
125+
"kind": "test",
126+
"payload": <test>
127+
}
128+
129+
<event-record> ::= {
130+
<version>,
131+
"kind": "event",
132+
"payload": <event>
133+
}
134+
```
135+
136+
### Tests
137+
138+
Test records represent individual test functions and test suites. Test records
139+
are passed through the record stream **before** most events.
140+
141+
<!--
142+
If a test record represents a parameterized test function whose inputs are
143+
enumerable and can be independently replayed, the test record will include an
144+
additional `"testCases"` field describing the individual test cases.
145+
-->
146+
147+
```
148+
<test> ::= <test-suite> | <test-function>
149+
150+
<test-suite> ::= {
151+
"kind": "suite",
152+
"name": <string>, ; the unformatted, unqualified type name
153+
["displayName": <string>,] ; the user-supplied custom display name
154+
"sourceLocation": <source-location>, ; where the test suite is defined
155+
"id": <test-id>,
156+
}
157+
158+
<test-function> ::= {
159+
"kind": "function",
160+
"name": <string>, ; the unformatted function name
161+
["displayName": <string>,] ; the user-supplied custom display name
162+
"sourceLocation": <source-location>, ; where the test is defined
163+
"id": <test-id>,
164+
"isParameterized": <bool> ; is this a parameterized test function or not?
165+
}
166+
167+
<test-id> ::= <string> ; an opaque string representing the test case
168+
```
169+
170+
<!--
171+
TODO: define a round-trippable format for a test case ID
172+
["testCases": <array:test-case>] ; if "isParameterized": true and the inputs
173+
; are enumerable, all test case IDs,
174+
; otherwise not present
175+
176+
<test-case> ::= {
177+
"id": <string>, ; an opaque string representing the test case
178+
"displayName": <string> ; a string representing the corresponding Swift value
179+
}
180+
```
181+
-->
182+
183+
### Events
184+
185+
Event records represent things that can happen during testing. They include
186+
information about the event such as when it occurred and where in the test
187+
source it occurred. They also include a `"messages"` field that contains
188+
sufficient information to display the event in a human-readable format.
189+
190+
```
191+
<event> ::= {
192+
"kind": <event-kind>,
193+
"instant": <instant>, ; when the event occurred
194+
["issue": <issue>,] ; the recorded issue (if "kind" is "issueRecorded")
195+
"messages": <array:message>,
196+
["testID": <test-id>,]
197+
}
198+
199+
<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
200+
"issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" |
201+
"runEnded" ; additional event kinds may be added in the future
202+
203+
<issue> ::= {
204+
"isKnown": <bool>, ; is this a known issue or not?
205+
["sourceLocation": <source-location>,] ; where the issue occurred, if known
206+
}
207+
208+
<message> ::= {
209+
"symbol": <message-symbol>,
210+
"text": <string>, ; the human-readable text of this message
211+
}
212+
213+
<message-symbol> ::= "default" | "skip" | "pass" | "passWithKnownIssue" | "fail"
214+
"difference" | "warning" | "details"
215+
```
216+
217+
<!--
218+
["testID": <test-id>,
219+
["testCase": <test-case>]]
220+
-->

‎Sources/Testing/EntryPoints/ABIEntryPoint.swift

+7-60
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,8 @@
1616
/// - argumentsJSON: A buffer to memory representing the JSON encoding of an
1717
/// instance of `__CommandLineArguments_v0`. If `nil`, a new instance is
1818
/// created from the command-line arguments to the current process.
19-
/// - eventHandler: An event handler to which is passed a buffer to memory
20-
/// representing each event and its context, as with ``Event/Handler``, but
21-
/// encoded as JSON.
19+
/// - recordHandler: A JSON record handler to which is passed a buffer to
20+
/// memory representing each record as described in `ABI/JSON.md`.
2221
///
2322
/// - Returns: The result of invoking the testing library. The type of this
2423
/// value is subject to change.
@@ -31,7 +30,7 @@
3130
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
3231
public typealias ABIEntryPoint_v0 = @Sendable (
3332
_ argumentsJSON: UnsafeRawBufferPointer?,
34-
_ eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
33+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
3534
) async -> CInt
3635

3736
/// Get the entry point to the testing library used by tools that want to remain
@@ -49,75 +48,23 @@ public typealias ABIEntryPoint_v0 = @Sendable (
4948
///
5049
/// The returned function can be thought of as equivalent to
5150
/// `swift test --experimental-event-stream-output` except that, instead of
52-
/// streaming events to a named pipe or file, it streams them to a callback.
51+
/// streaming JSON records to a named pipe or file, it streams them to an
52+
/// in-process callback.
5353
///
5454
/// - Warning: This function's signature and the structure of its JSON inputs
5555
/// and outputs have not been finalized yet.
5656
@_cdecl("swt_copyABIEntryPoint_v0")
5757
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
5858
public func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
5959
let result = UnsafeMutablePointer<ABIEntryPoint_v0>.allocate(capacity: 1)
60-
result.initialize { argumentsJSON, eventHandler in
60+
result.initialize { argumentsJSON, recordHandler in
6161
let args = try! argumentsJSON.map { argumentsJSON in
6262
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
6363
}
6464

65-
let eventHandler = eventHandlerForStreamingEvents_v0(to: eventHandler)
65+
let eventHandler = try! eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
6666
return await entryPoint(passing: args, eventHandler: eventHandler)
6767
}
6868
return .init(result)
6969
}
7070
#endif
71-
72-
// MARK: - Experimental event streaming
73-
74-
#if canImport(Foundation) && (!SWT_NO_FILE_IO || !SWT_NO_ABI_ENTRY_POINT)
75-
/// A type containing an event snapshot and snapshots of the contents of an
76-
/// event context suitable for streaming over JSON.
77-
///
78-
/// This function is not part of the public interface of the testing library.
79-
/// External adopters are not necessarily written in Swift and are expected to
80-
/// decode the JSON produced for this type in implementation-specific ways.
81-
struct EventAndContextSnapshot {
82-
/// A snapshot of the event.
83-
var event: Event.Snapshot
84-
85-
/// A snapshot of the event context.
86-
var eventContext: Event.Context.Snapshot
87-
}
88-
89-
extension EventAndContextSnapshot: Codable {}
90-
91-
/// Create an event handler that encodes events as JSON and forwards them to an
92-
/// ABI-friendly event handler.
93-
///
94-
/// - Parameters:
95-
/// - eventHandler: The event handler to forward events to. See
96-
/// ``ABIEntryPoint_v0`` for more information.
97-
///
98-
/// - Returns: An event handler.
99-
///
100-
/// The resulting event handler outputs data as JSON. For each event handled by
101-
/// the resulting event handler, a JSON object representing it and its
102-
/// associated context is created and is passed to `eventHandler`. These JSON
103-
/// objects are guaranteed not to contain any ASCII newline characters (`"\r"`
104-
/// or `"\n"`).
105-
///
106-
/// Note that `_eventHandlerForStreamingEvents_v0(toFileAtPath:)` calls this
107-
/// function and performs additional postprocessing before writing JSON data.
108-
func eventHandlerForStreamingEvents_v0(
109-
to eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
110-
) -> Event.Handler {
111-
return { event, context in
112-
let snapshot = EventAndContextSnapshot(
113-
event: Event.Snapshot(snapshotting: event),
114-
eventContext: Event.Context.Snapshot(snapshotting: context)
115-
)
116-
try? JSON.withEncoding(of: snapshot) { eventAndContextJSON in
117-
eventAndContextJSON.withUnsafeBytes { eventAndContextJSON in
118-
eventHandler(eventAndContextJSON)
119-
}
120-
}
121-
}
122-
}
123-
#endif

0 commit comments

Comments
 (0)