Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define and implement a strawman stable JSON schema for output. #383

Merged
merged 5 commits into from
May 10, 2024
Merged

Define and implement a strawman stable JSON schema for output. #383

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f0a4206
Start defining and implementing a stable JSON schema for output.
grynspan Apr 23, 2024
a2aa5a0
Incorporate feedback
grynspan May 9, 2024
9cbcae9
Incorporate feedback
grynspan May 10, 2024
e1d6ad0
Incorporate feedback
grynspan May 10, 2024
7172d9c
Incorporate feedback about test functions
grynspan May 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 220 additions & 0 deletions Documentation/ABI/JSON.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
# JSON schema

<!--
This source file is part of the Swift.org open source project

Copyright (c) 2024 Apple Inc. and the Swift project authors
Licensed under Apache License v2.0 with Runtime Library Exception

See https://swift.org/LICENSE.txt for license information
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
-->

This document outlines the JSON schemas used by the testing library for its ABI
entry point and for the `--experimental-event-stream-output` command-line
argument. For more information about the ABI entry point, see the documentation
for [ABI.EntryPoint_v0](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint_v0&type=code).

> [!WARNING]
> This JSON schema is still being developed and is subject to any and all
> changes including removal from the package.

## Modified Backus-Naur form

This schema is expressed using a modified Backus-Naur syntax. `{`, `}`, `:`, and
`,` represent their corresponding JSON tokens. `\n` represents an ASCII newline
character.

The order of keys in JSON objects is not normative. Whitespace in this schema is
not normative; it is present to help the reader understand the content of the
various JSON objects in the schema. The event stream is output using the JSON
Lines format and does not include newline characters (except **one** at the end
of the `<output-record-line>` rule.)

Trailing commas in JSON objects and arrays are only to be included where
syntactically valid.

### Common data types

`<string>` and `<number>` are defined as in JSON. `<array:T>` represents an
array (also defined as in JSON) whose elements all follow rule `<T>`.

```
<bool> ::= true | false ; as in JSON

<source-location> ::= {
"fileID": <string>, ; the Swift file ID of the file
"line": <number>,
"column": <number>,
}

<version> ::= "version": 0 ; will be incremented as the format changes

<boolean> ::= true | false ; boolean value as in JSON
```

<!--
TODO: implement input/configuration

### Configuration

A single configuration is passed into the testing library prior to running any
tests and, as the name suggests, configures the test run. The configuration is
encoded as a single [JSON Lines](https://jsonlines.org) value.

```
<configuration-record> ::= {
<version>,
"kind": "configuration",
"payload": <configuration>
}

<configuration> ::= {
["verbosity": <number>,] ; 0 is the default; higher means more verbose output
; while negative values mean quieter output.
["filters": <array:test-filter>,] ; how to filter the tests in the test run
["parallel": <bool>,] ; whether to enable parallel testing (on by default)
; more TBD
}

<test-filter> ::= <test-filter-tag> | <test-filter-id>

<test-filter-action> ::= "include" | "exclude"

<test-filter-tag> ::= {
"action": <test-filter-action>,
"tags": <array:string>, ; the names of tags to include
"operator": <test-filter-tag-operator> ; how to combine the values in "tags"
}

<test-filter-tag-operator> ::= "any" | "all"

<test-filter-id> ::= {
"action": <test-filter-action>,
"id": <test-id> ; the ID of the test to filter in/out
}
```
-->

### Streams

A stream consists of a sequence of values encoded as [JSON Lines](https://jsonlines.org).
A single instance of `<output-stream>` is defined per test process and can be
accessed by passing `--experimental-event-stream-output` to the test executable
created by `swift build --build-tests`.

```
<output-stream> ::= <output-record>\n | <output-record>\n <output-stream>
```

### Records

Records represent the values produced on a stream. Each record is encoded on a
single line and can be decoded independently of other lines. If a decoder
encounters a record whose `"kind"` field is unrecognized, the decoder should
ignore that line.

```
<output-record> ::= <metadata-record> | <test-record> | <event-record>

<metadata-record> ::= {
<version>,
"kind": "metadata",
"payload": <metadata>
}

<test-record> ::= {
<version>,
"kind": "test",
"payload": <test>
}

<event-record> ::= {
<version>,
"kind": "event",
"payload": <event>
}
```

### Metadata

Metadata records are reserved for future use.

```
<metadata> ::= {
; unspecified JSON object content
}
```

### Tests

Test records represent individual test functions and test suites. Test records
are passed through the record stream **before** most events.

<!--
If a test record represents a parameterized test function whose inputs are
enumerable and can be independently replayed, the test record will include an
additional `"testCases"` field describing the individual test cases.
-->

```
<test> ::= {
"kind": <test-kind>,
"name": <string>, ; the unformatted function or non-qualified type name
["displayName": <string>,] ; the user-supplied custom display name
"sourceLocation": <source-location>, ; where the test is defined
"id": <test-id>,
}

<test-kind> ::= "suite" | "function" | "parameterizedFunction"

<test-id> ::= <string> ; an opaque string representing the test case
```

<!--
TODO: define a round-trippable format for a test case ID
["testCases": <array:test-case>] ; if kind is "parameterizedFunction" and
; the inputs are enumerable, all test case
; IDs, otherwise not present

<test-case> ::= {
"id": <string>, ; an opaque string representing the test case
"displayName": <string> ; a string representing the corresponding Swift value
}
```
-->

### Events

Event records represent things that can happen during testing. They include
information about the event such as when it occurred and where in the test
source it occurred. They also include a `"messages"` field that contains
sufficient information to display the event in a human-readable format.

```
<event> ::= {
"kind": <event-kind>,
["sourceLocation": <source-location>,]
"timestamp": <number>, ; floating-point seconds since test epoch
"timestampSince1970": <number>, ; floating-point seconds since UNIX epoch
"messages": <array:message>,
["testID": <test-id>,]
}

<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
"issueRecorded" | "knownIssueRecorded" | "testCaseEnded" | "testEnded" |
"testSkipped" | "runEnded" ; additional event kinds may be added in the future

<message> ::= {
"symbol": <message-symbol>,
"text": <string>, ; the human-readable text of this message
}

<message-symbol> ::= "default" | "skip" | "pass" | "passWithKnownIssue" | "fail"
"difference" | "warning" | "details"
```

<!--
["testID": <test-id>,
["testCase": <test-case>]]
-->
67 changes: 7 additions & 60 deletions Sources/Testing/EntryPoints/ABIEntryPoint.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@
/// - argumentsJSON: A buffer to memory representing the JSON encoding of an
/// instance of `__CommandLineArguments_v0`. If `nil`, a new instance is
/// created from the command-line arguments to the current process.
/// - eventHandler: An event handler to which is passed a buffer to memory
/// representing each event and its context, as with ``Event/Handler``, but
/// encoded as JSON.
/// - recordHandler: A JSON record handler to which is passed a buffer to
/// memory representing each record as described in `ABI/JSON.md`.
///
/// - Returns: The result of invoking the testing library. The type of this
/// value is subject to change.
Expand All @@ -31,7 +30,7 @@
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
public typealias ABIEntryPoint_v0 = @Sendable (
_ argumentsJSON: UnsafeRawBufferPointer?,
_ eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
) async -> CInt

/// Get the entry point to the testing library used by tools that want to remain
Expand All @@ -49,75 +48,23 @@ public typealias ABIEntryPoint_v0 = @Sendable (
///
/// The returned function can be thought of as equivalent to
/// `swift test --experimental-event-stream-output` except that, instead of
/// streaming events to a named pipe or file, it streams them to a callback.
/// streaming JSON records to a named pipe or file, it streams them to an
/// in-process callback.
///
/// - Warning: This function's signature and the structure of its JSON inputs
/// and outputs have not been finalized yet.
@_cdecl("swt_copyABIEntryPoint_v0")
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
public func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
let result = UnsafeMutablePointer<ABIEntryPoint_v0>.allocate(capacity: 1)
result.initialize { argumentsJSON, eventHandler in
result.initialize { argumentsJSON, recordHandler in
let args = try! argumentsJSON.map { argumentsJSON in
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
}

let eventHandler = eventHandlerForStreamingEvents_v0(to: eventHandler)
let eventHandler = try! eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
return await entryPoint(passing: args, eventHandler: eventHandler)
}
return .init(result)
}
#endif

// MARK: - Experimental event streaming

#if canImport(Foundation) && (!SWT_NO_FILE_IO || !SWT_NO_ABI_ENTRY_POINT)
/// A type containing an event snapshot and snapshots of the contents of an
/// event context suitable for streaming over JSON.
///
/// This function is not part of the public interface of the testing library.
/// External adopters are not necessarily written in Swift and are expected to
/// decode the JSON produced for this type in implementation-specific ways.
struct EventAndContextSnapshot {
/// A snapshot of the event.
var event: Event.Snapshot

/// A snapshot of the event context.
var eventContext: Event.Context.Snapshot
}

extension EventAndContextSnapshot: Codable {}

/// Create an event handler that encodes events as JSON and forwards them to an
/// ABI-friendly event handler.
///
/// - Parameters:
/// - eventHandler: The event handler to forward events to. See
/// ``ABIEntryPoint_v0`` for more information.
///
/// - Returns: An event handler.
///
/// The resulting event handler outputs data as JSON. For each event handled by
/// the resulting event handler, a JSON object representing it and its
/// associated context is created and is passed to `eventHandler`. These JSON
/// objects are guaranteed not to contain any ASCII newline characters (`"\r"`
/// or `"\n"`).
///
/// Note that `_eventHandlerForStreamingEvents_v0(toFileAtPath:)` calls this
/// function and performs additional postprocessing before writing JSON data.
func eventHandlerForStreamingEvents_v0(
to eventHandler: @escaping @Sendable (_ eventAndContextJSON: UnsafeRawBufferPointer) -> Void
) -> Event.Handler {
return { event, context in
let snapshot = EventAndContextSnapshot(
event: Event.Snapshot(snapshotting: event),
eventContext: Event.Context.Snapshot(snapshotting: context)
)
try? JSON.withEncoding(of: snapshot) { eventAndContextJSON in
eventAndContextJSON.withUnsafeBytes { eventAndContextJSON in
eventHandler(eventAndContextJSON)
}
}
}
}
#endif
Loading