Skip to content

Commit f0a4206

Browse files
grynspangrynspan
committed
Start defining and implementing a stable JSON schema for output.
1 parent a8f42e0 commit f0a4206

17 files changed

+879
-125
lines changed

‎Documentation/ABI/JSON.md

+222
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
# JSON schema
2+
3+
<!--
4+
This source file is part of the Swift.org open source project
5+
6+
Copyright (c) 2024 Apple Inc. and the Swift project authors
7+
Licensed under Apache License v2.0 with Runtime Library Exception
8+
9+
See https://swift.org/LICENSE.txt for license information
10+
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
11+
-->
12+
13+
This document outlines the JSON schemas used by the testing library for its ABI
14+
entry point and for the `--experimental-event-stream-output` command-line
15+
argument. For more information about the ABI entry point, see the documentation
16+
for [ABI.EntryPoint_v0](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint_v0&type=code).
17+
18+
> [!WARNING]
19+
> This JSON schema is still being developed and is subject to any and all
20+
> changes including removal from the package.
21+
22+
## Modified Backus-Naur form
23+
24+
This schema is expressed using a modified Backus-Naur syntax. `{`, `}`, `:`, and
25+
`,` represent their corresponding JSON tokens. `\n` represents an ASCII newline
26+
character.
27+
28+
The order of keys in JSON objects is not normative. Whitespace in this schema is
29+
not normative; it is present to help the reader understand the content of the
30+
various JSON objects in the schema. The event stream is output using the JSON
31+
Lines format and does not include newline characters (except **one** at the end
32+
of the `<output-record-line>` rule.)
33+
34+
Trailing commas in JSON objects and arrays are only to be included where
35+
syntactically valid.
36+
37+
### Common data types
38+
39+
`<string>` and `<number>` are defined as in JSON. `<array:T>` represents an
40+
array (also defined as in JSON) whose elements all follow rule `<T>`.
41+
42+
```
43+
<bool> ::= true | false ; as in JSON
44+
45+
<source-location> ::= {
46+
"fileID": <string>, ; the Swift file ID of the file
47+
"line": <number>,
48+
"column": <number>,
49+
}
50+
51+
<version> ::= "version": 0 ; will be incremented as the format changes
52+
53+
<boolean> ::= true | false ; boolean value as in JSON
54+
```
55+
56+
<!--
57+
TODO: implement input/configuration
58+
59+
### Configuration
60+
61+
A single configuration is passed into the testing library prior to running any
62+
tests and, as the name suggests, configures the test run. The configuration is
63+
encoded as a single [JSON Lines](https://jsonlines.org) value.
64+
65+
```
66+
<configuration-record> ::= {
67+
<version>,
68+
"kind": "configuration",
69+
"payload": <configuration>
70+
}
71+
72+
<configuration> ::= {
73+
["verbosity": <number>,] ; 0 is the default; higher means more verbose output
74+
; while negative values mean quieter output.
75+
["filters": <array:test-filter>,] ; how to filter the tests in the test run
76+
["parallel": <bool>,] ; whether to enable parallel testing (on by default)
77+
; more TBD
78+
}
79+
80+
<test-filter> ::= <test-filter-tag> | <test-filter-id>
81+
82+
<test-filter-action> ::= "include" | "exclude"
83+
84+
<test-filter-tag> ::= {
85+
"action": <test-filter-action>,
86+
"tags": <array:string>, ; the names of tags to include
87+
"operator": <test-filter-tag-operator> ; how to combine the values in "tags"
88+
}
89+
90+
<test-filter-tag-operator> ::= "any" | "all"
91+
92+
<test-filter-id> ::= {
93+
"action": <test-filter-action>,
94+
"id": <test-id> ; the ID of the test to filter in/out
95+
}
96+
```
97+
-->
98+
99+
### Streams
100+
101+
A stream consists of a sequence of values encoded as [JSON Lines](https://jsonlines.org).
102+
A single instance of `<output-stream>` is defined per test process and can be
103+
accessed by passing `--experimental-event-stream-output` to the test executable
104+
created by `swift build --build-tests`.
105+
106+
```
107+
<output-stream> ::= <output-record>\n | <output-record>\n <output-stream>
108+
```
109+
110+
### Records
111+
112+
Records represent the values produced on a stream. Each record is encoded on a
113+
single line and can be decoded independently of other lines. If a decoder
114+
encounters a record whose `"kind"` field is unrecognized, the decoder should
115+
ignore that line.
116+
117+
```
118+
<output-record> ::= <metadata-record> | <test-record> | <event-record>
119+
120+
<metadata-record> ::= {
121+
<version>,
122+
"kind": "metadata",
123+
"payload": <metadata>
124+
}
125+
126+
<test-record> ::= {
127+
<version>,
128+
"kind": "test",
129+
"payload": <test>
130+
}
131+
132+
<event-record> ::= {
133+
<version>,
134+
"kind": "event",
135+
"payload": <event>
136+
}
137+
```
138+
139+
### Metadata
140+
141+
Metadata records are reserved for future use.
142+
143+
```
144+
<metadata> ::= {
145+
; unspecified JSON object content
146+
}
147+
```
148+
149+
### Tests
150+
151+
Test records represent individual test functions and test suites. Test records
152+
are passed through the record stream **before** most events.
153+
154+
<!--
155+
If a test record represents a parameterized test function whose inputs are
156+
enumerable and can be independently replayed, the test record will include an
157+
additional `"testCases"` field describing the individual test cases.
158+
-->
159+
160+
```
161+
<test> ::= {
162+
"kind": <test-kind>,
163+
"name": <string>, ; the unformatted function or non-qualified type name
164+
["displayName": <string>,] ; the user-supplied custom display name
165+
"sourceLocation": <source-location>, ; where the test is defined
166+
"id": <test-id>,
167+
}
168+
169+
<test-kind> ::= "suite" | "function" | "parameterizedFunction"
170+
171+
<test-id> ::= <string> ; an opaque string representing the test case
172+
```
173+
174+
<!--
175+
TODO: define a round-trippable format for a test case ID
176+
["testCases": <array:test-case>] ; if kind is "parameterizedFunction" and
177+
; the inputs are enumerable, all test case
178+
; IDs, otherwise not present
179+
180+
<test-case> ::= {
181+
"id": <string>, ; an opaque string representing the test case
182+
"displayName": <string> ; a string representing the corresponding Swift value
183+
}
184+
```
185+
-->
186+
187+
### Events
188+
189+
Event records represent things that can happen during testing. They include
190+
information about the event such as when it occurred and where in the test
191+
source it occurred. They also include a `"messages"` field that contains
192+
sufficient information to display the event in a human-readable format.
193+
194+
```
195+
<event> ::= {
196+
"kind": <event-kind>,
197+
["sourceLocation": <source-location>,]
198+
"timestamp": <number>, ; floating-point seconds since test epoch
199+
"timestampSince1970": <number>, ; floating-point seconds since UNIX epoch
200+
"messages": <array:message>,
201+
["testID": <test-id>,]
202+
}
203+
204+
<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
205+
"issueRecorded" | "knownIssueRecorded" | "testCaseEnded" | "testEnded" |
206+
"testSkipped" | "runEnded" ; additional event kinds may be added in the future
207+
208+
<message> ::= {
209+
"symbol": <message-symbol>,
210+
"text": <string>, ; the human-readable text of this message
211+
["markdown": <string>] ; if available/desired/whatever, Markdown encoding
212+
; the same string as "text"
213+
}
214+
215+
<message-symbol> ::= "default" | "skip" | "pass" | "passWithKnownIssue" | "fail"
216+
"difference" | "warning" | "details"
217+
```
218+
219+
<!--
220+
["testID": <test-id>,
221+
["testCase": <test-case>]]
222+
-->

‎Sources/Testing/EntryPoints/ABIEntryPoint.swift

+7-60
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,8 @@
1616
/// - argumentsJSON: A buffer to memory representing the JSON encoding of an
1717
/// instance of `__CommandLineArguments_v0`. If `nil`, a new instance is
1818
/// created from the command-line arguments to the current process.
19-
/// - eventHandler: An event handler to which is passed a buffer to memory
20-
/// representing each event and its context, as with ``Event/Handler``, but
21-
/// encoded as JSON.
19+
/// - recordHandler: A JSON record handler to which is passed a buffer to
20+
/// memory representing each record as described in `ABI/JSON.md`.
2221
///
2322
/// - Returns: The result of invoking the testing library. The type of this
2423
/// value is subject to change.
@@ -31,7 +30,7 @@
3130
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
3231
public typealias ABIEntryPoint_v0 = @Sendable (
3332
_ argumentsJSON: UnsafeRawBufferPointer?,
34-
_ eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
33+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
3534
) async -> CInt
3635

3736
/// Get the entry point to the testing library used by tools that want to remain
@@ -49,75 +48,23 @@ public typealias ABIEntryPoint_v0 = @Sendable (
4948
///
5049
/// The returned function can be thought of as equivalent to
5150
/// `swift test --experimental-event-stream-output` except that, instead of
52-
/// streaming events to a named pipe or file, it streams them to a callback.
51+
/// streaming JSON records to a named pipe or file, it streams them to an
52+
/// in-process callback.
5353
///
5454
/// - Warning: This function's signature and the structure of its JSON inputs
5555
/// and outputs have not been finalized yet.
5656
@_cdecl("swt_copyABIEntryPoint_v0")
5757
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
5858
public func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
5959
let result = UnsafeMutablePointer<ABIEntryPoint_v0>.allocate(capacity: 1)
60-
result.initialize { argumentsJSON, eventHandler in
60+
result.initialize { argumentsJSON, recordHandler in
6161
let args = try! argumentsJSON.map { argumentsJSON in
6262
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
6363
}
6464

65-
let eventHandler = eventHandlerForStreamingEvents_v0(to: eventHandler)
65+
let eventHandler = try! eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
6666
return await entryPoint(passing: args, eventHandler: eventHandler)
6767
}
6868
return .init(result)
6969
}
7070
#endif
71-
72-
// MARK: - Experimental event streaming
73-
74-
#if canImport(Foundation) && (!SWT_NO_FILE_IO || !SWT_NO_ABI_ENTRY_POINT)
75-
/// A type containing an event snapshot and snapshots of the contents of an
76-
/// event context suitable for streaming over JSON.
77-
///
78-
/// This function is not part of the public interface of the testing library.
79-
/// External adopters are not necessarily written in Swift and are expected to
80-
/// decode the JSON produced for this type in implementation-specific ways.
81-
struct EventAndContextSnapshot {
82-
/// A snapshot of the event.
83-
var event: Event.Snapshot
84-
85-
/// A snapshot of the event context.
86-
var eventContext: Event.Context.Snapshot
87-
}
88-
89-
extension EventAndContextSnapshot: Codable {}
90-
91-
/// Create an event handler that encodes events as JSON and forwards them to an
92-
/// ABI-friendly event handler.
93-
///
94-
/// - Parameters:
95-
/// - eventHandler: The event handler to forward events to. See
96-
/// ``ABIEntryPoint_v0`` for more information.
97-
///
98-
/// - Returns: An event handler.
99-
///
100-
/// The resulting event handler outputs data as JSON. For each event handled by
101-
/// the resulting event handler, a JSON object representing it and its
102-
/// associated context is created and is passed to `eventHandler`. These JSON
103-
/// objects are guaranteed not to contain any ASCII newline characters (`"\r"`
104-
/// or `"\n"`).
105-
///
106-
/// Note that `_eventHandlerForStreamingEvents_v0(toFileAtPath:)` calls this
107-
/// function and performs additional postprocessing before writing JSON data.
108-
func eventHandlerForStreamingEvents_v0(
109-
to eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
110-
) -> Event.Handler {
111-
return { event, context in
112-
let snapshot = EventAndContextSnapshot(
113-
event: Event.Snapshot(snapshotting: event),
114-
eventContext: Event.Context.Snapshot(snapshotting: context)
115-
)
116-
try? JSON.withEncoding(of: snapshot) { eventAndContextJSON in
117-
eventAndContextJSON.withUnsafeBytes { eventAndContextJSON in
118-
eventHandler(eventAndContextJSON)
119-
}
120-
}
121-
}
122-
}
123-
#endif

0 commit comments

Comments
 (0)