Skip to content

Test: ensure swift-[build|test] emit expected number of fatal error on console #8076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

bkhouri
Copy link
Contributor

@bkhouri bkhouri commented Oct 25, 2024

Add an automated tests that ensures swift-build and swift-test emit the expected amount of error: fatalError message on the console output.

Motivation:

Issue #6605 reported error: fatalError was emitted twice to the console output, and has since been fixed. We should ensure the issue does not regress.

Modifications:

Add an automated tests that verifies swift-build does not emit a error: fatalError in stdout or stderr when a building a test that does not compile.

Result:

❯ swift test --filter TestGetNumberOfMatches --filter testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation
Building for debugging...
[10/10] Linking SwiftPMPackageTests
Build complete! (9.88s)
Test Suite 'Selected tests' started at 2024-10-25 13:59:47.463.
Test Suite 'SwiftPMPackageTests.xctest' started at 2024-10-25 13:59:47.465.
Test Suite 'BuildCommandTests' started at 2024-10-25 13:59:47.465.
Test Case '-[CommandsTests.BuildCommandTests testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation]' started.
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:685 | INFO | GIVEN we have a Swift Package that has a fatalError building the tests
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:687 | INFO | WHEN swift-build --build-tests is executed
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:689 | INFO | THEN I expect a failure
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:699 | DEBUG | stdout: "[0/1] Planning build\nBuilding for debugging...\n[0/7] /private/var/folders/1b/57vwczd16h733tf_gzsx7d8c0000gn/T/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary.JjD0VE/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary/.build/arm64-apple-macosx/debug/TypeLibraryPackageTests.derived/runner.swift\n[1/7] Write sources\n[4/7] Write swift-version-3DF09919E8911743.txt\n[6/9] Emitting module TypeLibrary\n[7/9] Compiling TypeLibrary TypeLibrary.swift\n[8/11] Emitting module TypeLibraryTests\n[9/11] Compiling TypeLibraryTests TypeLibraryTests.swift\n/private/var/folders/1b/57vwczd16h733tf_gzsx7d8c0000gn/T/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary.JjD0VE/Miscellaneous_Errors_FatalErrorInSingleXCTest_TypeLibrary/Tests/TypeLibraryTests/TypeLibraryTests.swift:10:15: error: division by zero\n 8 | func testExample() throws {\n 9 |     let x = 0\n10 |     let y = 1 / x\n   |               `- error: division by zero\n11 |     print(y)\n12 | }\n"
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:700 | DEBUG | stderr: ""
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:701 | DEBUG | number stdout matches: 0
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:702 | DEBUG | number stderr matches: 0
Emitted by Test | /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Tests/CommandsTests/BuildCommandTests.swift:704 | INFO | AND a fatal error message is not printed to the console
Test Case '-[CommandsTests.BuildCommandTests testFatalErrorDisplayedOnlyOnceWhenSingleXCTestHasFatalErrorInBuildCompilaation]' passed (3.564 seconds).
Test Suite 'BuildCommandTests' passed at 2024-10-25 13:59:51.029.
         Executed 1 test, with 0 failures (0 unexpected) in 3.564 (3.564) seconds
Test Suite 'TestGetNumberOfMatches' started at 2024-10-25 13:59:51.029.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnEmptyStringZeroTimes]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnEmptyStringZeroTimes]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringUsingTripleDoubleQuotesZeroTimes]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringUsingTripleDoubleQuotesZeroTimes]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringWithNeLineCharacterZeroTimes]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptyMultilineStringWithNeLineCharacterZeroTimes]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptySingleLineStringZeroTimes]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testEmptyStringMatchesOnNonEmptySingleLineStringZeroTimes]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingAndTrailingNewLineMatchesOnMultiLineWithOneOccurencesReturnsOne]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingAndTrailingNewLineMatchesOnMultiLineWithOneOccurencesReturnsOne]' passed (0.000 seconds).
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingNewLineMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' started.
Test Case '-[_InternalTestSupportTests.TestGetNumberOfMatches testfatalErrorWithLeadingNewLineMatchesOnMultiLineWithTwoOccurencesReturnsTwo]' passed (0.000 seconds).
Test Suite 'TestGetNumberOfMatches' passed at 2024-10-25 13:59:51.030.
         Executed 7 tests, with 0 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'SwiftPMPackageTests.xctest' passed at 2024-10-25 13:59:51.030.
         Executed 8 tests, with 0 failures (0 unexpected) in 3.565 (3.566) seconds
Test Suite 'Selected tests' passed at 2024-10-25 13:59:51.030.
         Executed 8 tests, with 0 failures (0 unexpected) in 3.565 (3.567) seconds
◇ Test run started.
↳ Testing Library Version: 102 (arm64e-apple-macos13.0)
✔ Test run with 0 tests passed after 0.001 seconds.```

@plemarquand
Copy link
Contributor

Thanks for the contribution! A few comments, and I noticed the test fixture is missing.

@bkhouri bkhouri force-pushed the t/main/gh6605_add_test_fatal_error_occurs_onces_in_output branch from 2e93797 to 8cf6b88 Compare October 25, 2024 20:32
Copy link
Contributor Author

@bkhouri bkhouri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. I have revised the PR.

Comment on lines 121 to 123



Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting is off here, would you mind running the formatter on newly added files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran swift format format --in-place Tests/_InternalTestSupportTests/Misc.swift on the file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still see stray whitespaces surrounding curly braces. Unfortunately, swift-format is not great at following our formatting style and lacks required customizations to specify it. We're using https://github.com/nicklockwood/SwiftFormat in SwiftPM instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatting all touched file using https://github.com/nicklockwood/SwiftFormat resulted in many changes. so I only formatted this file.

@bkhouri bkhouri force-pushed the t/main/gh6605_add_test_fatal_error_occurs_onces_in_output branch from 8cf6b88 to d281225 Compare October 28, 2024 18:49
Copy link
Contributor

@plemarquand plemarquand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending @MaxDesiatov signing off on formatting

@bkhouri bkhouri force-pushed the t/main/gh6605_add_test_fatal_error_occurs_onces_in_output branch from d281225 to f972479 Compare October 29, 2024 22:16
Copy link
Contributor

@MaxDesiatov MaxDesiatov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci test windows

…n console

Add an automated tests that ensures swift-build and swift-test emit the
expected amount of `error: fatalError` message on the console output.
@bkhouri bkhouri force-pushed the t/main/gh6605_add_test_fatal_error_occurs_onces_in_output branch from f972479 to 8edd10e Compare October 30, 2024 20:53
@plemarquand
Copy link
Contributor

@swift-ci test

@bkhouri bkhouri changed the title Test: ensure swift-build does not emit fatal error on console Test: ensure swift-[build|test] emit expected number of fatal error on console Oct 31, 2024
@bkhouri
Copy link
Contributor Author

bkhouri commented Oct 31, 2024

The Swift Test Linux Platform build failure appears to be unrelated to my change

[1999/2062] Compiling Commands DescribedPackage.swift
error: compile command failed due to signal 6 (use -v to see invocation)
swift-frontend: /home/build-user/llvm-project/llvm/include/llvm/Support/Casting.h:109: static bool llvm::isa_impl_cl<clang::RecordDecl, const clang::Decl *>::doit(const From *) [To = clang::RecordDecl, From = const clang::Decl *]: Assertion `Val && "isa<> used on a null pointer"' failed.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
Stack dump:
0.	Program arguments: /home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/bin/swift-frontend -frontend -c /home/build-user/swiftpm/Sources/Commands/CommandWorkspaceDelegate.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/APIDiff.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/AddDependency.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/AddProduct.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/AddTarget.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/AddTargetDependency.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ArchiveSource.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/CompletionCommand.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ComputeChecksum.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Config.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Describe.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/DumpCommands.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/EditCommands.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Format.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Init.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Install.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Learn.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/PluginCommand.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ResetCommands.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Resolve.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ShowDependencies.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ShowExecutables.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/SwiftPackageCommand.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/ToolsVersionCommand.swift /home/build-user/swiftpm/Sources/Commands/PackageCommands/Update.swift /home/build-user/swiftpm/Sources/Commands/Snippets/Card.swift /home/build-user/swiftpm/Sources/Commands/Snippets/CardEvent.swift /home/build-user/swiftpm/Sources/Commands/Snippets/CardStack.swift /home/build-user/swiftpm/Sources/Commands/Snippets/Cards/SnippetCard.swift /home/build-user/swiftpm/Sources/Commands/Snippets/Cards/SnippetGroupCard.swift /home/build-user/swiftpm/Sources/Commands/Snippets/Cards/TopCard.swift /home/build-user/swiftpm/Sources/Commands/Snippets/Colorful.swift /home/build-user/swiftpm/Sources/Commands/SwiftBuildCommand.swift /home/build-user/swiftpm/Sources/Commands/SwiftRunCommand.swift /home/build-user/swiftpm/Sources/Commands/SwiftTestCommand.swift /home/build-user/swiftpm/Sources/Commands/Utilities/APIDigester.swift /home/build-user/swiftpm/Sources/Commands/Utilities/DOTManifestSerializer.swift /home/build-user/swiftpm/Sources/Commands/Utilities/DependenciesSerializer.swift /home/build-user/swiftpm/Sources/Commands/Utilities/DescribedPackage.swift /home/build-user/swiftpm/Sources/Commands/Utilities/MermaidPackageSerializer.swift -primary-file /home/build-user/swiftpm/Sources/Commands/Utilities/MultiRootSupport.swift /home/build-user/swiftpm/Sources/Commands/Utilities/PlainTextEncoder.swift /home/build-user/swiftpm/Sources/Commands/Utilities/PluginDelegate.swift /home/build-user/swiftpm/Sources/Commands/Utilities/SymbolGraphExtract.swift /home/build-user/swiftpm/Sources/Commands/Utilities/TestingSupport.swift /home/build-user/swiftpm/Sources/Commands/Utilities/XCTEvents.swift -emit-dependencies-path /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/Commands.build/MultiRootSupport.d -emit-reference-dependencies-path /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/Commands.build/MultiRootSupport.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/Modules -enable-testing -g -debug-info-format=dwarf -dwarf-version=4 -module-cache-path /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/ModuleCache -static -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -empty-abi-descriptor -resource-dir /home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/lib/swift -enable-anonymous-context-mangled-names -file-compilation-dir /home/build-user/swiftpm -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-system/Sources/CSystem/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-system/Sources/CSystem/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-tools-support-core/Sources/TSCclibc/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-tools-support-core/Sources/TSCclibc/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/Sources/SPMSQLite3/module.modulemap -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/Yams/Sources/CYaml/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/Yams/Sources/CYaml/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-driver/Sources/CSwiftScan/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-driver/Sources/CSwiftScan/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-llbuild/products/libllbuild/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/products/libllbuild/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llbuildNinja.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/Ninja/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llbuildBasic.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/Basic/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llvmSupport.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/llvm/Support/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llvmDemangle.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/llvm/Demangle/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llbuildBuildSystem.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/BuildSystem/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/llbuildCore.build/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-llbuild/lib/Core/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-crypto/Sources/CCryptoBoringSSLShims/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-crypto/Sources/CCryptoBoringSSLShims/include -Xcc -fmodule-map-file=/home/build-user/swiftpm/.build/checkouts/swift-crypto/Sources/CCryptoBoringSSL/include/module.modulemap -Xcc -I -Xcc /home/build-user/swiftpm/.build/checkouts/swift-crypto/Sources/CCryptoBoringSSL/include -Xcc -fPIC -Xcc -g -Xcc -fno-omit-frame-pointer -module-name Commands -package-name swiftpm -in-process-plugin-server-path /home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/lib/swift/host/libSwiftInProcPluginServer.so -plugin-path /home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/lib/swift/host/plugins -plugin-path /home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/local/lib/swift/host/plugins -parse-as-library -o /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/Commands.build/MultiRootSupport.swift.o -index-store-path /home/build-user/swiftpm/.build/x86_64-unknown-linux-gnu/debug/index/store -index-system-modules
1.	Swift version 6.1-dev (LLVM f4b733c38006ec1, Swift f4bea5f7c0a7aa5)
2.	Compiling with effective version 5.10
3.	While reading from module 'FoundationXML', builder version '6.1(6.1)/Swift version 6.1-dev (LLVM f4b733c38006ec1, Swift f4bea5f7c0a7aa5)', built from source against SDK , non-resilient, loaded from '/home/build-user/latest_toolchain/swift-DEVELOPMENT-SNAPSHOT-2024-10-27-a-ubuntu20.04/usr/lib/swift/linux/FoundationXML.swiftmodule/x86_64-unknown-linux-gnu.swiftmodule'
4.	While deserializing '_NSXMLParserExternalEntityWithURL' (FuncDecl @ 1270116) in 'FoundationXML'
5.	    ...decl is named '_NSXMLParserExternalEntityWithURL(_:urlStr:identifier:context:originalLoaderFunction:)'
6.	While deserializing 'originalLoaderFunction' (ParamDecl @ 1404256) in 'FoundationXML'
7.	    ...decl is named 'originalLoaderFunction'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend  0x0000561b30f4a767
1  swift-frontend  0x0000561b30f4846e

@plemarquand
Copy link
Contributor

@swift-ci please test Linux

@bkhouri
Copy link
Contributor Author

bkhouri commented Nov 1, 2024

I believe swiftlang/swift#77309 was the underlying build issue and was merged about 6 hours ago. Can we re-trigger the failed build?

@bnbarham
Copy link
Contributor

bnbarham commented Nov 1, 2024

@swift-ci please test Windows platform

@bnbarham
Copy link
Contributor

bnbarham commented Nov 1, 2024

@swift-ci please test Linux platform

@dschaefer2 dschaefer2 merged commit f3e3fb6 into swiftlang:main Nov 4, 2024
5 checks passed
@bkhouri bkhouri deleted the t/main/gh6605_add_test_fatal_error_occurs_onces_in_output branch November 4, 2024 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants