Skip to content

Commit f3e3fb6

Browse files
authored
Test: ensure swift-[build|test] emit expected number of fatal error on console (#8076)
Add an automated tests that ensures swift-build and swift-test emit the expected amount of `error: fatalError` message on the console output. ### Motivation: Issue #6605 reported `error: fatalError` was emitted twice to the console output, and has since been fixed. We should ensure the issue does not regress. ### Modifications: Add an automated tests that verifies `swift-build` does not emit a `error: fatalError` in `stdout` or `stderr` when a building a test that does not compile. ### Result: ``` ❯ swift test --filter TestGetNumberOfMatches --filter testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation Building for debugging... [10/10] Linking SwiftPMPackageTests Build complete! (9.88s) Test Suite 'Selected tests' started at 2024-10-25 13:59:47.463. Test Suite 'SwiftPMPackageTests.xctest' started at 2024-10-25 13:59:47.465. Test Suite 'BuildCommandTests' started at 2024-10-25 13:59:47.465. Test Case '-[CommandsTests.BuildCommandTests testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation]' started. Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:685 | INFO | GIVEN we have a Swift Package that has a fatalError building the tests Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:687 | INFO | WHEN swift-build --build-tests is executed Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:689 | INFO | THEN I expect a failure Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:699 | DEBUG | stdout: "[0/1] Planning build\nBuilding for debugging...\n[0/7] /private/var/folders/1b/57vwczd16h733tf_gzsx7d8c0000gn/T/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary.JjD0VE/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary/.build/arm64-apple-macosx/debug/TypeLibraryPackageTests.derived/runner.swift\n[1/7] Write sources\n[4/7] Write swift-version-3DF09919E8911743.txt\n[6/9] Emitting module TypeLibrary\n[7/9] Compiling TypeLibrary TypeLibrary.swift\n[8/11] Emitting module TypeLibraryTests\n[9/11] Compiling TypeLibraryTests TypeLibraryTests.swift\n/private/var/folders/1b/57vwczd16h733tf_gzsx7d8c0000gn/T/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary.JjD0VE/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary/Tests/TypeLibraryTests/TypeLibraryTests.swift:10:15: error: division by zero\n 8 | func testExample() throws {\n 9 | let x = 0\n10 | let y = 1 / x\n | `- error: division by zero\n11 | print(y)\n12 | }\n" Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:700 | DEBUG | stderr: "" Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:701 | DEBUG | number stdout matches: 0 Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:702 | DEBUG | number stderr matches: 0 Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:704 | INFO | AND a fatal error message is not printed to the console Test Case '-[CommandsTests.BuildCommandTests testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation]' passed (3.564 seconds). Test Suite 'BuildCommandTests' passed at 2024-10-25 13:59:51.029. Executed 1 test, with 0 failures (0 unexpected) in 3.564 (3.564) seconds Test Suite 'TestGetNumberOfMatches' started at 2024-10-25 13:59:51.029. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnEmptyStringZeroTimes]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnEmptyStringZeroTimes]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringUsingTripleDoubleQuotesZeroTimes]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringUsingTripleDoubleQuotesZeroTimes]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringWithNeLineCharacterZeroTimes]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringWithNeLineCharacterZeroTimes]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptySingleLineStringZeroTimes]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptySingleLineStringZeroTimes]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingAndTrailingNewLineMatchesOnMultiLineWithOneOccurencesReturnsOne]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingAndTrailingNewLineMatchesOnMultiLineWithOneOccurencesReturnsOne]' passed (0.000 seconds). Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingNewLineMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' started. Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingNewLineMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' passed (0.000 seconds). Test Suite 'TestGetNumberOfMatches' passed at 2024-10-25 13:59:51.030. Executed 7 tests, with 0 failures (0 unexpected) in 0.001 (0.001) seconds Test Suite 'SwiftPMPackageTests.xctest' passed at 2024-10-25 13:59:51.030. Executed 8 tests, with 0 failures (0 unexpected) in 3.565 (3.566) seconds Test Suite 'Selected tests' passed at 2024-10-25 13:59:51.030. Executed 8 tests, with 0 failures (0 unexpected) in 3.565 (3.567) seconds ◇ Test run started. ↳ Testing Library Version: 102 (arm64e-apple-macos13.0) ✔ Test run with 0 tests passed after 0.001 seconds.```
1 parent 5e953fe commit f3e3fb6

File tree

9 files changed

+245
-1
lines changed

9 files changed

+245
-1
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.DS_Store
2+
/.build
3+
/Packages
4+
xcuserdata/
5+
DerivedData/
6+
.swiftpm/configuration/registries.json
7+
.swiftpm/xcode/package.xcworkspace/contents.xcworkspacedata
8+
.netrc
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
// swift-tools-version:5.9
2+
// The swift-tools-version declares the minimum version of Swift required to build this package.
3+
4+
import PackageDescription
5+
6+
let package = Package(
7+
name: "TypeLibrary",
8+
products: [
9+
// Products define the executables and libraries a package produces, making them visible to other packages.
10+
.library(
11+
name: "TypeLibrary",
12+
targets: ["TypeLibrary"]),
13+
],
14+
targets: [
15+
// Targets are the basic building blocks of a package, defining a module or a test suite.
16+
// Targets can depend on other targets in this package and products from dependencies.
17+
.target(
18+
name: "TypeLibrary"),
19+
.testTarget(
20+
name: "TypeLibraryTests",
21+
dependencies: ["TypeLibrary"]
22+
),
23+
]
24+
)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
// The Swift Programming Language
2+
// https://docs.swift.org/swift-book
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
func testExample() throws {
2+
let x = 0
3+
let y = 1 / x
4+
print(y)
5+
}

Package.swift

+6-1
Original file line numberDiff line numberDiff line change
@@ -873,7 +873,12 @@ if ProcessInfo.processInfo.environment["SWIFTCI_DISABLE_SDK_DEPENDENT_TESTS"] ==
873873
"Basics",
874874
]
875875
),
876-
876+
.testTarget(
877+
name: "_InternalTestSupportTests",
878+
dependencies: [
879+
"_InternalTestSupport"
880+
]
881+
),
877882
.testTarget(
878883
name: "CommandsTests",
879884
dependencies: [

Sources/_InternalTestSupport/misc.swift

+6
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
//
1111
//===----------------------------------------------------------------------===//
1212

13+
import Foundation
1314
import Basics
1415
import struct Foundation.URL
1516
#if os(macOS)
@@ -465,3 +466,8 @@ extension PackageIdentity: @retroactive ExpressibleByStringInterpolation {}
465466
extension AbsolutePath: @retroactive ExpressibleByStringLiteral {}
466467
extension AbsolutePath: @retroactive ExpressibleByStringInterpolation {}
467468
#endif
469+
470+
public func getNumberOfMatches(of match: String, in value: String) -> Int {
471+
guard match.count != 0 else { return 0 }
472+
return value.ranges(of: match).count
473+
}

Tests/CommandsTests/BuildCommandTests.swift

+38
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
//
1111
//===----------------------------------------------------------------------===//
1212

13+
import Foundation
1314
import Basics
1415
@testable import Commands
1516
@testable import CoreCommands
@@ -678,4 +679,41 @@ final class BuildCommandTests: CommandsTestCase {
678679
XCTAssertGreaterThan(codeCovFiles.count, 0)
679680
}
680681
}
682+
683+
func testFatalErrorDisplayedCorrectNumberOfTimesWhenSingleXCTestHasFatalErrorInBuildCompilation() async throws {
684+
// Test for GitHub Issue #6605
685+
// GIVEN we have a Swift Package that has a fatalError building the tests
686+
#if compiler(>=6)
687+
let expected = 0
688+
#else
689+
let expected = 1
690+
#endif
691+
try await fixture(name: "Miscellaneous/Errors/FatalErrorInSingleXCTest/TypeLibrary") { fixturePath in
692+
// WHEN swift-build --build-tests is executed"
693+
await XCTAssertAsyncThrowsError(try await self.execute(["--build-tests"], packagePath: fixturePath)) { error in
694+
// THEN I expect a failure
695+
guard case SwiftPMError.executionFailure(_, let stdout, let stderr) = error else {
696+
XCTFail("Building the package was expected to fail, but it was successful")
697+
return
698+
}
699+
700+
let matchString = "error: fatalError"
701+
let stdoutMatches = getNumberOfMatches(of: matchString, in: stdout)
702+
let stderrMatches = getNumberOfMatches(of: matchString, in: stderr)
703+
let actualNumMatches = stdoutMatches + stderrMatches
704+
705+
// AND a fatal error message is printed \(expected) times
706+
XCTAssertEqual(
707+
actualNumMatches,
708+
expected,
709+
[
710+
"Actual (\(actualNumMatches)) is not as expected (\(expected))",
711+
"stdout: \(stdout.debugDescription)",
712+
"stderr: \(stderr.debugDescription)"
713+
].joined(separator: "\n")
714+
)
715+
}
716+
}
717+
}
718+
681719
}

Tests/CommandsTests/TestCommandTests.swift

+37
Original file line numberDiff line numberDiff line change
@@ -361,4 +361,41 @@ final class TestCommandTests: CommandsTestCase {
361361
XCTAssertNoMatch(stderr, .contains("No matching test cases were run"))
362362
}
363363
}
364+
365+
func testFatalErrorDisplayedCorrectNumberOfTimesWhenSingleXCTestHasFatalErrorInBuildCompilation() async throws {
366+
// Test for GitHub Issue #6605
367+
// GIVEN we have a Swift Package that has a fatalError building the tests
368+
#if compiler(>=6)
369+
let expected = 1
370+
#else
371+
let expected = 2
372+
#endif
373+
try await fixture(name: "Miscellaneous/Errors/FatalErrorInSingleXCTest/TypeLibrary") { fixturePath in
374+
// WHEN swift-test is executed
375+
await XCTAssertAsyncThrowsError(try await self.execute([], packagePath: fixturePath)) { error in
376+
// THEN I expect a failure
377+
guard case SwiftPMError.executionFailure(_, let stdout, let stderr) = error else {
378+
XCTFail("Building the package was expected to fail, but it was successful")
379+
return
380+
}
381+
382+
let matchString = "error: fatalError"
383+
let stdoutMatches = getNumberOfMatches(of: matchString, in: stdout)
384+
let stderrMatches = getNumberOfMatches(of: matchString, in: stderr)
385+
let actualNumMatches = stdoutMatches + stderrMatches
386+
387+
// AND a fatal error message is printed \(expected) times
388+
XCTAssertEqual(
389+
actualNumMatches,
390+
expected,
391+
[
392+
"Actual (\(actualNumMatches)) is not as expected (\(expected))",
393+
"stdout: \(stdout.debugDescription)",
394+
"stderr: \(stderr.debugDescription)"
395+
].joined(separator: "\n")
396+
)
397+
}
398+
}
399+
}
400+
364401
}
+119
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
import _InternalTestSupport
2+
import XCTest
3+
4+
final class TestGetNumberOfMatches: XCTestCase {
5+
func testEmptyStringMatchesOnEmptyStringZeroTimes() {
6+
let matchOn = ""
7+
let value = ""
8+
let expectedNumMatches = 0
9+
10+
let actual = getNumberOfMatches(of: matchOn, in: value)
11+
12+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
13+
}
14+
15+
func testEmptyStringMatchesOnNonEmptySingleLineStringZeroTimes() {
16+
let matchOn = ""
17+
let value = "This is a non-empty string"
18+
let expectedNumMatches = 0
19+
20+
let actual = getNumberOfMatches(of: matchOn, in: value)
21+
22+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
23+
}
24+
25+
func testEmptyStringMatchesOnNonEmptyMultilineStringWithNeLineCharacterZeroTimes() {
26+
let matchOn = ""
27+
let value = "This is a non-empty string\nThis is the second line"
28+
let expectedNumMatches = 0
29+
30+
let actual = getNumberOfMatches(of: matchOn, in: value)
31+
32+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
33+
}
34+
35+
func testEmptyStringMatchesOnNonEmptyMultilineStringUsingTripleDoubleQuotesZeroTimes() {
36+
let matchOn = ""
37+
let value = """
38+
This is a non-empty string
39+
This is the second line
40+
This is the third line
41+
"""
42+
let expectedNumMatches = 0
43+
44+
let actual = getNumberOfMatches(of: matchOn, in: value)
45+
46+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
47+
}
48+
49+
func testNonEmptyStringMatchesOnEmptyStringReturnsZero() {
50+
let matchOn = """
51+
This is a non-empty string
52+
This is the second line
53+
This is the third line
54+
"""
55+
let value = ""
56+
let expectedNumMatches = 0
57+
58+
let actual = getNumberOfMatches(of: matchOn, in: value)
59+
60+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
61+
}
62+
63+
func testfatalErrorMatchesOnMultiLineWithTwoOccurencesReturnsTwo() {
64+
let matchOn = "error: fatalError"
65+
let value = """
66+
> swift test 25/10/24 10:44:14
67+
Building for debugging...
68+
/Users/arandomuser/Documents/personal/repro-swiftpm-6605/Tests/repro-swiftpm-6605Tests/repro_swiftpm_6605Tests.swift:7:19: error: division by zero
69+
let y = 1 / x
70+
^
71+
error: fatalError
72+
73+
error: fatalError
74+
"""
75+
let expectedNumMatches = 2
76+
77+
let actual = getNumberOfMatches(of: matchOn, in: value)
78+
79+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
80+
}
81+
82+
func testfatalErrorWithLeadingNewLineMatchesOnMultiLineWithTwoOccurencesReturnsTwo() {
83+
let matchOn = "\nerror: fatalError"
84+
let value = """
85+
> swift test 25/10/24 10:44:14
86+
Building for debugging...
87+
/Users/arandomuser/Documents/personal/repro-swiftpm-6605/Tests/repro-swiftpm-6605Tests/repro_swiftpm_6605Tests.swift:7:19: error: division by zero
88+
let y = 1 / x
89+
^
90+
error: fatalError
91+
92+
error: fatalError
93+
"""
94+
let expectedNumMatches = 2
95+
96+
let actual = getNumberOfMatches(of: matchOn, in: value)
97+
98+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
99+
}
100+
101+
func testfatalErrorWithLeadingAndTrailingNewLineMatchesOnMultiLineWithOneOccurencesReturnsOne() {
102+
let matchOn = "\nerror: fatalError\n"
103+
let value = """
104+
> swift test 25/10/24 10:44:14
105+
Building for debugging...
106+
/Users/arandomuser/Documents/personal/repro-swiftpm-6605/Tests/repro-swiftpm-6605Tests/repro_swiftpm_6605Tests.swift:7:19: error: division by zero
107+
let y = 1 / x
108+
^
109+
error: fatalError
110+
111+
error: fatalError
112+
"""
113+
let expectedNumMatches = 1
114+
115+
let actual = getNumberOfMatches(of: matchOn, in: value)
116+
117+
XCTAssertEqual(actual, expectedNumMatches, "Actual is not as expected")
118+
}
119+
}

0 commit comments

Comments
 (0)