Support building on macOS to develop this project #8

kateinoigakukun · 2019-10-26T04:55:11Z

This pr supports to develop this project on macOS and apply some changes to track upstream llvm and swift changes.

Dumped projects hashes config

{
    "https-clone-pattern": "https://github.com/%s.git", 
    "ssh-clone-pattern": "[email protected]:%s.git", 
    "repos": {
        "sourcekit-lsp": {
            "remote": {
                "id": "apple/sourcekit-lsp"
            }
        }, 
        "swift-syntax": {
            "remote": {
                "id": "apple/swift-syntax"
            }
        }, 
        "cmake": {
            "platforms": [
                "Linux"
            ], 
            "remote": {
                "id": "KitWare/CMake"
            }
        }, 
        "swift-xcode-playground-support": {
            "remote": {
                "id": "apple/swift-xcode-playground-support"
            }
        }, 
        "swift-format": {
            "remote": {
                "id": "apple/swift-format"
            }
        }, 
        "swift-stress-tester": {
            "remote": {
                "id": "apple/swift-stress-tester"
            }
        }, 
        "llvm-project": {
            "remote": {
                "id": "swiftwasm/llvm-project"
            }
        }, 
        "llbuild": {
            "remote": {
                "id": "apple/swift-llbuild"
            }
        }, 
        "cmark": {
            "remote": {
                "id": "apple/swift-cmark"
            }
        }, 
        "swift-corelibs-foundation": {
            "remote": {
                "id": "apple/swift-corelibs-foundation"
            }
        }, 
        "indexstore-db": {
            "remote": {
                "id": "apple/indexstore-db"
            }
        }, 
        "swift-corelibs-xctest": {
            "remote": {
                "id": "apple/swift-corelibs-xctest"
            }
        }, 
        "ninja": {
            "remote": {
                "id": "ninja-build/ninja"
            }
        }, 
        "swift-integration-tests": {
            "remote": {
                "id": "apple/swift-integration-tests"
            }
        }, 
        "swiftpm": {
            "remote": {
                "id": "apple/swift-package-manager"
            }
        }, 
        "swift": {
            "remote": {
                "id": "swiftwasm/swift"
            }
        }, 
        "swift-corelibs-libdispatch": {
            "remote": {
                "id": "apple/swift-corelibs-libdispatch"
            }
        }, 
        "icu": {
            "remote": {
                "id": "unicode-org/icu"
            }
        }
    }, 
    "branch-schemes": {
        "wasm": {
            "repos": {
                "sourcekit-lsp": "c051601d77512c5ca7394da280c57b0914ff9baa", 
                "indexstore-db": "363c7044cca2dd01724fa45bd602c89a129a1b23", 
                "cmake": "skip", 
                "swift-xcode-playground-support": "8caffe97601cd20f0cfe4fb354e81541f1a7f576", 
                "swift-stress-tester": "e43cf5312b972a315e78561158dcca613663fdc0", 
                "llbuild": "68477be467721ebdc3ba1b302d5ca592b8eafea9", 
                "cmark": "bfa95d55b535fa178f75484b5e8f82ae3d8517af", 
                "swift-corelibs-foundation": "6d9af2044f7dc29b1609cd8680fb9684d14695ce", 
                "llvm-project": "52fb2be63b3d5591aa989d323d51940cc7c3e193", 
                "swiftpm": "1ccff6203cbcd801211a27fd45c59a0b58775a3b", 
                "swift-corelibs-xctest": "aada6467671a399093e4c8569376bfd73e68af57", 
                "swift-format": "a23dc14a4f0cc4c1975025a6a5a7b995cba9363f", 
                "swift-syntax": "0cba2138d6a54a804a41e42607f947ef32754dee", 
                "ninja": "b25c08bda4949192c69cea4cee057887341a2ffc", 
                "swift-integration-tests": "11f0f6e8b34ba9782b5841dbeaa207d0b4620152", 
                "swift": "4839621a9e61f705a223f902858907d23be1d3cd", 
                "swift-corelibs-libdispatch": "c992dacf3ca114806e6ac9ffc9113b19255be9fe", 
                "icu": "b3054b334d8cf84dcae12df7e1a4f1f5c0997445"
            }, 
            "aliases": [
                "wasm"
            ]
        }
    }
}

kateinoigakukun · 2019-10-26T04:56:12Z

CMakeLists.txt

+    # WebAssembly: hack: use llvm-ar for creating static libraries; Ubuntu's GNU ar doesn't work with wasm-ld
+    set(CMAKE_AR "${SWIFT_WASM_WASI_SDK_PATH}/bin/llvm-ar")
+  endif()
+


I moved set(CMAKE_AR ...) line to use prebuilt llvm-ar only on Linux.
In macOS environment, this hack is unnecessary.

kateinoigakukun · 2019-10-26T05:02:01Z

stdlib/public/runtime/SwiftRT-WASM.cpp

@@ -0,0 +1,11 @@
+//===--- SwiftRT-WASM.cpp --------------------------------------------------===//


I separated SwiftRT file to distinguish ELF and WASM.
No need to emit __start and __stop symbol because the runtime system which uses custom sections are disabled now and the latest clang raised runtime error to interpret __asm__((__section__))

kateinoigakukun · 2019-10-26T05:10:32Z

stdlib/public/core/StringStorage.swift

@@ -416,7 +416,7 @@ extension __StringStorage {
    let count = try initializer(buffer)

    let countAndFlags = CountAndFlags(mortalCount: count, isASCII: false)
-    #if arch(i386) || arch(arm)
+    #if arch(i386) || arch(arm) || arch(wasm32)


These lines are added by upstream Swift

MaxDesiatov · 2019-10-27T16:26:48Z

stdlib/public/core/StringStorage.swift

@@ -369,7 +369,7 @@ extension __StringStorage {
      __StringStorage.self,
      realCodeUnitCapacity._builtinWordValue, UInt8.self,
      1._builtinWordValue, Optional<_StringBreadcrumbs>.self)
-#if arch(i386) || arch(arm) || arch(wasm32)
+#if arch(i386) || arch(arm) || arch(wasm32) || arch(wasm32)


arch(wasm32) condition seems to be duplicated here

my bad 😅

MaxDesiatov

Overall the changes make sense, just a duplicate macro check is present, which can be cleaned up

kateinoigakukun · 2019-10-27T22:52:01Z

@MaxDesiatov Thanks for reviewing!

MaxDesiatov

@kateinoigakukun thanks for updating the PR, the changes seem legit. Would you mind sharing a ./swift/utils/build-script invocation with its arguments that you use to test this? Or do you build it as a part of swiftwasm-sdk?

kateinoigakukun · 2019-10-28T10:36:26Z

@MaxDesiatov I'm using that command

utils/build-script --debug --wasm \
  --skip-build-benchmarks \
  --extra-cmake-options=" \
    -DSWIFT_PRIMARY_VARIANT_SDK:STRING=WASM \
    -DSWIFT_PRIMARY_VARIANT_ARCH:STRING=wasm32 \
    -DSWIFT_OSX_x86_64_ICU_STATICLIB=TRUE \
    -DSWIFT_BUILD_SOURCEKIT=FALSE \
    -DSWIFT_ENABLE_SOURCEKIT_TESTS=FALSE \
  " \
  --stdlib-deployment-targets "wasm-wasm32" \
  --build-stdlib-deployment-targets "wasm-wasm32" \
  --llvm-targets-to-build "X86;WebAssembly" \
  --build-swift-dynamic-sdk-overlay false \
  --build-swift-static-sdk-overlay false \
  --wasm-wasi-sdk "$sourcedir/wasi-sdk" \
  --wasm-icu-uc "$sourcedir/icu_out/lib/libicuuc.a" \
  --wasm-icu-uc-include "$sourcedir/icu_out/include" \
  --wasm-icu-i18n "$sourcedir/icu_out/lib/libicui18n.a" \
  --wasm-icu-i18n-include "$sourcedir/icu_out/include" \
  --wasm-icu-data "todo-icu-data" \
  --build-swift-static-stdlib

MaxDesiatov · 2019-10-28T12:54:05Z

@kateinoigakukun unfortunately this doesn't build for me locally due to an error, a build log snippet is pasted below:

-- Wasm SDK:
--   Object File Format: WASM
--   Swift Standard Library Path: wasm
--   Architectures: wasm32
--   wasm32 triple: wasm32-unknown-unknown-wasm
--   wasm32 Path: /Users/maxd/Documents/swiftwasm-sdk/wasi-sdk/share/sysroot
--   wasm32 libc header path: /usr/include
--   wasm32 libc architecture specific header path: /usr/include/
--   wasm32 ICU i18n INCLUDE: /Users/maxd/Documents/swiftwasm-sdk/icu_out/include
--   wasm32 ICU i18n LIB: /Users/maxd/Documents/swiftwasm-sdk/icu_out/lib/libicui18n.a
--   wasm32 ICU unicode INCLUDE: /Users/maxd/Documents/swiftwasm-sdk/icu_out/include
--   wasm32 ICU unicode LIB: /Users/maxd/Documents/swiftwasm-sdk/icu_out/lib/libicuuc.a
-- 
-- Building host Swift tools for OSX x86_64
--   Build type:     Debug
--   Assertions:     TRUE
--   LTO:            
-- 
-- Building Swift standard library and overlays for SDKs: WASM
--   Build type:       Debug
--   Assertions:       TRUE
-- 
-- Building Swift runtime with:
--   Leak Detection Checker Entrypoints: FALSE
-- 
-- Found PythonInterp: /Users/maxd/.pyenv/shims/python (found version "2.7.16") 
-- Found LibXml2: /usr/lib/libxml2.dylib (found version "2.9.4") 
-- Looking for el_wgets
-- Looking for el_wgets - found
-- Looking for wait4
-- Looking for wait4 - found
-- Looking for proc_pid_rusage
-- Looking for proc_pid_rusage - found
-- Performing Test is_supported
-- Performing Test is_supported - Success
-- Performing Test CXX_SUPPORTS_GLOBAL_CONSTRUCTORS_WARNING
-- Performing Test CXX_SUPPORTS_GLOBAL_CONSTRUCTORS_WARNING - Success
-- Performing Test CXX_SUPPORTS_EXIT_TIME_DESTRUCTORS_WARNING
-- Performing Test CXX_SUPPORTS_EXIT_TIME_DESTRUCTORS_WARNING - Success
-- Performing Test CXX_SUPPORTS_DEFAULT_HIDDEN_VISIBILITY
-- Performing Test CXX_SUPPORTS_DEFAULT_HIDDEN_VISIBILITY - Success
-- Differentiable programming standard library additions enabled.
-- Swift version: 5.1.1
-- Swift vendor: 
-- Swift compiler version: 
-- Embedded clang compiler version: 
CMake Warning (dev) at /usr/local/Cellar/cmake/3.15.4/share/cmake/Modules/CheckIncludeFiles.cmake:120 (message):
  Policy CMP0075 is not set: Include file check macros honor
  CMAKE_REQUIRED_LIBRARIES.  Run "cmake --help-policy CMP0075" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  CMAKE_REQUIRED_LIBRARIES is set to:

    proc

  For compatibility with CMake 3.11 and below this check is ignoring it.
Call Stack (most recent call first):
  tools/CMakeLists.txt:2 (check_include_files)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Looking for include file xpc/xpc.h
-- Looking for include file xpc/xpc.h - found
CMake Warning at docs/CMakeLists.txt:24 (message):
  Unable to find sphinx-build program.  Not building docs


-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CLANG_VERSION_MAJOR
    CLANG_VERSION_MINOR
    CLANG_VERSION_PATCH
    CMAKE_LIBTOOL
    COVERAGE_DB
    LIBDISPATCH_CMAKE_BUILD_TYPE
    SWIFTLIB_DEPLOYMENT_VERSION_XCTEST_IOS
    SWIFT_BUILD_EXAMPLES
    SWIFT_DARWIN_XCRUN_TOOLCHAIN
    SWIFT_HOST_TRIPLE
    SWIFT_PATH_TO_LIBDISPATCH_BUILD
    SWIFT_WASM_wasm32_ICU_DATA


-- Build files have been written to: /Users/maxd/Documents/swiftwasm-sdk/build/Ninja-DebugAssert/swift-macosx-x86_64
+ popd
~/Documents/swiftwasm-sdk/swift
+ /usr/local/bin/cmake --build /Users/maxd/Documents/swiftwasm-sdk/build/Ninja-DebugAssert/swift-macosx-x86_64 -- -j12 all swift-test-stdlib-wasm-wasm32 swift-test-stdlib-wasm-wasm32
ninja: error: '/Users/maxd/Documents/swiftwasm-sdk/icu_out/lib/libicuuc.a', needed by 'lib/swift/wasm/wasm32/libswiftCore.dylib', missing and no known rule to make it
utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

Is there any step missing? Maybe I need to prebuild icu before running utils/build-script?

MaxDesiatov · 2019-10-28T12:56:56Z

@kateinoigakukun you could also merge #10 into your branch, which I hope would reproduce this error on our CI

MaxDesiatov · 2019-10-30T16:50:14Z

@kateinoigakukun looks like I got #10 past that error, would you be able to merge that gh-actions-macos into your branch of this PR to run the CI job on it? Thanks!

…support-darwin-develop

…w value

In this case, SWIFT_PRIMARY_VARIANT_ARCH is used but host SDK was not PRIMARY_SDK.

This reverts commit 5e3916e.

kateinoigakukun · 2019-11-02T15:26:07Z

Build succeeded on both Linux and macOS 🎉

MaxDesiatov

In the future it would be great to refactor the CI scripts: move common invocations to a separate .sh file, make it download the latest wasi SDK etc, but for now this seems legit 👍

* Add simple GitHub Actions config for macOS * Add brew install to the macOS GH action * Add update-checkout call to macOS GH action * Export sourcedir shell variable in macOS GH action * [WASM] Use prebuilt llvm-ar only on Linux * [WASM] Added new Object format WASM to distinguish from ELF * [WASM] Separate SwiftRT from ELF * [WASM] Add preprocessor condition for wasm32 to track latest upstream * Add wasi-sdk and icu install steps to the script * [WASM] Set LIBC_INCLUDE_DIRECTORY without CACHE attribute to force new value * [WASM] Adding share path to point correct directory * [WASM] Add Linux CI job * [WASM] Checkout branch after checking out Swift repos * [WASM] Use release build to reduce artifact size * [WASM] Use SWIFT_PRIMARY_VARIANT_SDK instead of HOST_SDK In this case, SWIFT_PRIMARY_VARIANT_ARCH is used but host SDK was not PRIMARY_SDK. * [WASM] Remove ICU_STATICLIB on Linux * [WASM] Output build log verbose * [WASM] Set ICU lib directory instead of archive * [WASM] Sort build options * Revert "[WASM] Use SWIFT_PRIMARY_VARIANT_SDK instead of HOST_SDK"

kateinoigakukun commented Oct 26, 2019

View reviewed changes

kateinoigakukun changed the title ~~Katei/support darwin develop~~ Support macOS to develop this project Oct 26, 2019

kateinoigakukun changed the title ~~Support macOS to develop this project~~ Support building on macOS to develop this project Oct 26, 2019

kateinoigakukun commented Oct 26, 2019

View reviewed changes

MaxDesiatov reviewed Oct 27, 2019

View reviewed changes

MaxDesiatov requested changes Oct 27, 2019

View reviewed changes

kateinoigakukun force-pushed the katei/support-darwin-develop branch from 4839621 to e2edb55 Compare October 27, 2019 22:50

MaxDesiatov reviewed Oct 28, 2019

View reviewed changes

MaxDesiatov added 3 commits October 28, 2019 10:47

Add simple GitHub Actions config for macOS

7b563cc

Add brew install to the macOS GH action

544c068

Add update-checkout call to macOS GH action

1de0656

MaxDesiatov self-assigned this Oct 28, 2019

MaxDesiatov assigned kateinoigakukun and unassigned MaxDesiatov Oct 28, 2019

Export sourcedir shell variable in macOS GH action

3998824

kateinoigakukun and others added 5 commits October 30, 2019 05:44

[WASM] Use prebuilt llvm-ar only on Linux

f4c2205

[WASM] Added new Object format WASM to distinguish from ELF

1b3daf9

[WASM] Separate SwiftRT from ELF

08549c7

[WASM] Add preprocessor condition for wasm32 to track latest upstream

cad6531

Add wasi-sdk and icu install steps to the script

19ca6b1

kateinoigakukun force-pushed the katei/support-darwin-develop branch from e2edb55 to cad6531 Compare October 30, 2019 22:32

kateinoigakukun added 4 commits October 30, 2019 22:33

Merge remote-tracking branch 'swiftwasm/gh-actions-macos' into katei/…

eb76d92

…support-darwin-develop

[WASM] Set LIBC_INCLUDE_DIRECTORY without CACHE attribute to force ne…

f09c370

…w value

[WASM] Adding share path to point correct directory

258f25e

[WASM] Add Linux CI job

8d90ac3

[WASM] Checkout branch after checking out Swift repos

8c87d29

kateinoigakukun force-pushed the katei/support-darwin-develop branch from 18f7542 to 8c87d29 Compare October 31, 2019 01:38

kateinoigakukun added 7 commits October 31, 2019 13:26

[WASM] Use release build to reduce artifact size

72cd605

[WASM] Use SWIFT_PRIMARY_VARIANT_SDK instead of HOST_SDK

5e3916e

In this case, SWIFT_PRIMARY_VARIANT_ARCH is used but host SDK was not PRIMARY_SDK.

[WASM] Remove ICU_STATICLIB on Linux

a089ff2

[WASM] Output build log verbose

a19bb18

[WASM] Set ICU lib directory instead of archive

b320192

[WASM] Sort build options

6e88e0a

Revert "[WASM] Use SWIFT_PRIMARY_VARIANT_SDK instead of HOST_SDK"

df43f21

This reverts commit 5e3916e.

kateinoigakukun force-pushed the katei/support-darwin-develop branch from 72c09a7 to df43f21 Compare November 2, 2019 11:05

MaxDesiatov approved these changes Nov 2, 2019

View reviewed changes

MaxDesiatov merged commit fb939ab into swiftwasm:swiftwasm Nov 2, 2019

MaxDesiatov mentioned this pull request Nov 2, 2019

Add simple GitHub Actions config for macOS #10

Closed

MaxDesiatov added the continuous integration Improvements to continuous integration label Nov 17, 2019

kateinoigakukun deleted the katei/support-darwin-develop branch January 28, 2020 12:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support building on macOS to develop this project #8

Support building on macOS to develop this project #8

kateinoigakukun commented Oct 26, 2019 •

edited

Loading

kateinoigakukun Oct 26, 2019

kateinoigakukun Oct 26, 2019

kateinoigakukun Oct 26, 2019

MaxDesiatov Oct 27, 2019

kateinoigakukun Oct 27, 2019

MaxDesiatov left a comment

kateinoigakukun commented Oct 27, 2019

MaxDesiatov left a comment •

edited

Loading

kateinoigakukun commented Oct 28, 2019

MaxDesiatov commented Oct 28, 2019

MaxDesiatov commented Oct 28, 2019

MaxDesiatov commented Oct 30, 2019

kateinoigakukun commented Nov 2, 2019

MaxDesiatov left a comment

		@@ -0,0 +1,11 @@
		//===--- SwiftRT-WASM.cpp --------------------------------------------------===//

Support building on macOS to develop this project #8

Support building on macOS to develop this project #8

Conversation

kateinoigakukun commented Oct 26, 2019 • edited Loading

kateinoigakukun Oct 26, 2019

Choose a reason for hiding this comment

kateinoigakukun Oct 26, 2019

Choose a reason for hiding this comment

kateinoigakukun Oct 26, 2019

Choose a reason for hiding this comment

MaxDesiatov Oct 27, 2019

Choose a reason for hiding this comment

kateinoigakukun Oct 27, 2019

Choose a reason for hiding this comment

MaxDesiatov left a comment

Choose a reason for hiding this comment

kateinoigakukun commented Oct 27, 2019

MaxDesiatov left a comment • edited Loading

Choose a reason for hiding this comment

kateinoigakukun commented Oct 28, 2019

MaxDesiatov commented Oct 28, 2019

MaxDesiatov commented Oct 28, 2019

MaxDesiatov commented Oct 30, 2019

kateinoigakukun commented Nov 2, 2019

MaxDesiatov left a comment

Choose a reason for hiding this comment

kateinoigakukun commented Oct 26, 2019 •

edited

Loading

MaxDesiatov left a comment •

edited

Loading