Skip to content

Add a Swift parser library that is written in Swift #616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 22, 2022
Merged

Conversation

ahoppen
Copy link
Member

@ahoppen ahoppen commented Aug 22, 2022

This starts an effort to implement the parser for the Swift compiler in Swift itself.

For more information, please see the announcement forum post at: https://forums.swift.org/t/a-new-swift-parser-for-swiftsyntax/59813

This starts an effort to implement the parser for the Swift compiler in Swift itself.

Co-authored-by: Robert Widmann <[email protected]>
Co-authored-by: Doug Gregor <[email protected]>
Co-authored-by: Rintaro Ishizaki <[email protected]>
@ahoppen
Copy link
Member Author

ahoppen commented Aug 22, 2022

@swift-ci Please test

@etcwilde
Copy link
Contributor

@compnerd, if you have a cycle or two, it's probably worth making sure this works on Windows.

@compnerd
Copy link
Member

Doesn't seem to build on Windows (which is a regression). At the very least we will need swiftlang/swift-docc-plugin#26.

@CodaFi
Copy link
Contributor

CodaFi commented Aug 22, 2022

The docc plugin is not required for this repository in general, and is not a dependency for this PR specifically. Docc support is built into Xcode, I am not sure why we have a plugin dependency as well.

@compnerd
Copy link
Member

I couldn't answer why we have that. We should invert the condition so that it is not used by default.

@DougGregor
Copy link
Member

I'm fine with removing the DocC plugin dependency or inverting the condition. We shouldn't land this in a form that breaks the Windows build.

@CodaFi
Copy link
Contributor

CodaFi commented Aug 22, 2022

Let's strike it for now. We shouldn't have merged it without verifying that it worked for Windows

#617

Copy link
Member

@compnerd compnerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This at least allows this PR to be built on Windows with:

swift build -Xcc -IC:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\include

Note that building this requires a locally built toolchain as there was a fix for SPM that is needed that has not had a new toolchain snapshot built due to other failures introduced by the rebranch.

@DougGregor
Copy link
Member

@swift-ci please test

1 similar comment
@DougGregor
Copy link
Member

@swift-ci please test

@DougGregor
Copy link
Member

@swift-ci please test

@@ -0,0 +1,70 @@
## Overview
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(not a blocker) The existence of this file triggers a warning, at least with 5.7:

warning: 'swift-syntax': found 1 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    Sources/SwiftParser/README.md

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll merge and we'll fix it up afterward. Thanks!

@wu-s-john
Copy link

Nice. Swift would really benefit from code generating it's own code rather than C++.

@broken-bytes
Copy link

I can assure you that this does currently not work on Windows. I just tried using this framework in an SPM project on Windows.

I am getting the following error:

[41/88] Compiling SwiftSyntax AtomicCounter.swift
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.      Program arguments: C:\\Library\\Developer\\Toolchains\\unknown-Asserts-development.xctoolchain\\usr\\bin\\swiftc.exe -frontend -c H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\AbsolutePosition.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\AtomicCounter.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\CNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\IncrementalParseTransition.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\RawSyntax.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourceLength.swift -primary-file H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourceLocation.swift -primary-file H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourcePresence.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\Syntax.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxChildren.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxClassifier.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxData.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxOtherNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxVerifier.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\Utils.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\_SyntaxParserInterop.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\Misc.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxAnyVisitor.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxBaseNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxBuilders.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxClassification.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxCollections.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxEnum.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxFactory.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxKind.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxRewriter.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxTraits.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxVisitor.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\TokenKind.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\Trivia.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxDeclNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxExprNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxPatternNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxStmtNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxTypeNodes.swift -emit-module-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftmodule -emit-module-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftmodule -emit-module-doc-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftdoc -emit-module-doc-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftdoc -emit-module-source-info-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftsourceinfo -emit-module-source-info-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftsourceinfo -emit-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.d -emit-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.d -emit-reference-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.swiftdeps -emit-reference-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.swiftdeps -target x86_64-unknown-windows-msvc -disable-objc-interop -sdk C:\\Library\\Developer\\Platforms\\Windows.platform\\Developer\\SDKs\\Windows.sdk -I H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug -I C:\\Library\\Developer\\Platforms\\Windows.platform\\Developer\\Library\\XCTest-development\\usr\\lib\\swift\\windows\\x86_64 -enable-testing -g -module-cache-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\ModuleCache -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -Xcc -fmodule-map-file=H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\_CSwiftSyntax.build\\module.modulemap -Xcc -I -Xcc H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\_CSwiftSyntax\\include -autolink-library oldnames -autolink-library msvcrt -Xcc -D_MT -Xcc -D_DLL -parse-as-library -module-name SwiftSyntax -o H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.swift.o -o H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.swift.o -index-store-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\index\\store -index-system-modules
1.      compnerd.org Swift version 5.6.3 (swift-5.6.3-RELEASE)
2.      Compiling with the current language version
3.      While evaluating request IRGenRequest(IR Generation for file "H:\Projects\Cyanite-Rewrite\cyanite-engine\cyanite\api\.build\checkouts\swift-syntax\Sources\SwiftSyntax\SourceLocation.swift")
 #0 0x000001b55a432310
 #1 0x00007ff61db030d9 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x38530d9)
 #2 0x00007ff61da6f0d3 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x37bf0d3)
 #3 0x00007ff61da2365f (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x377365f)
 #4 0x00007ff61e3ba844 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x410a844)
 #5 0x00007ff61e3c0fe0 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4110fe0)
 #6 0x00007ff61a812bda (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x562bda)
 #7 0x00007ff61a75ee1b (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4aee1b)
 #8 0x00007ff61a74cabf (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x49cabf)
 #9 0x00007ff61a756e06 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4a6e06)
#10 0x00007ff61a744f90 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x494f90)
#11 0x00007ff61a750bd8 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4a0bd8)
#12 0x00007ff61a3766c0 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc66c0)
#13 0x00007ff61a37705e (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc705e)
#14 0x00007ff61a375cbb (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc5cbb)
#15 0x00007ff61a378097 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc8097)
#16 0x00007ff61a3398ec (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x898ec)
#17 0x00007ff61a33951e (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x8951e)
#18 0x00007ff61f015fbc (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4d65fbc)
#19 0x00007ff8939d54e0 (C:\Windows\System32\KERNEL32.DLL+0x154e0)
#20 0x00007ff8948c485b (C:\Windows\SYSTEM32\ntdll.dll+0x485b)

@compnerd
Copy link
Member

I can assure you that this does currently not work on Windows. I just tried using this framework in an SPM project on Windows.

Please use a build from main. At the very least, a recent snapshot.

@broken-bytes
Copy link

broken-bytes commented Sep 13, 2022

I used the most recent version.

dependencies: [
        .package(
            url: "https://github.com/apple/swift-syntax.git", from: "0.50600.1"),
    ],

As far as I am concerned, this is the most recent snapshot

@compnerd
Copy link
Member

5.6 should be buildable - https://github.com/compnerd/swift-build/releases/tag/swift-format-5.6-RELEASE was built and would pull in swift-syntax 5.6. I wonder if the .1 has anything to do with it 🤔

@DougGregor DougGregor deleted the parser-library branch September 13, 2022 16:49
@DougGregor
Copy link
Member

@broken-bytes If you're looking to try the new parser that's the subject of this pull request, you'll need main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants