Skip to content

Commit 3536ad1

Browse files
compnerdcompnerd
committed
Windows: add an image for building the toolchain
This prepares a docker container with Visual Studio Build Tools, Windows SDK, Python, and Git. It additionally sets up the required symbolic links into the Visual Studio installation (assuming that the sources are checked out to the documented canonical location of `S:\SourceCache`). This should enable a user to run something along the lines of: ``` docker create --compress -t swift:swift-ci . docker run -v C:\data:S:\ -it swift-ci ```
1 parent 594ab4b commit 3536ad1

File tree

5 files changed

+208
-0
lines changed

5 files changed

+208
-0
lines changed

‎swift-ci/README.md

+107
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,113 @@ docker run \
3535
/bin/bash -lc "cp -r /source/* /home/build-user/; ./swift/utils/build-script --preset buildbot_linux install_destdir=/home/build-user/swift-install installable_package=/home/build-user/swift-DEVELOPMENT-SNAPSHOT-$(date +'%F')-a.tar.gz"
3636
```
3737

38+
## Quick Start for Windows Development
39+
40+
The Windows Docker image will setup an enviornment with Python, Visual Studio
41+
Build Tools, and Git. It is setup to assume that the sources will be available
42+
in `S:\SourceCache`.
43+
44+
### Building and Tagging the image
45+
46+
```powershell
47+
cd master\windows\10.0.19044.1706
48+
docker image build --compress -t swift:swiftci .
49+
```
50+
51+
### Running the image
52+
53+
```powershell
54+
docker run --rm -it -v %UserProfile%\data:S: swift:swiftci
55+
```
56+
57+
### Building the Toolchain
58+
59+
While we can build the toolchain in the containerized environment, the sources
60+
are expected to reside on the host and is passed into the docker container as a
61+
volume. The rest of automation expects that the Sources reside under a
62+
directory with the name `SourceCache`.
63+
64+
#### Clone the Sources
65+
66+
```cmd
67+
md %UserProfile%\data\SourceCache
68+
cd %UserProfile%\data\SourceCache
69+
70+
git clone -b stable/20220426 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/llvm-project
71+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift
72+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-cmark cmark
73+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-experimental-string-processing
74+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-libdispatch
75+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-foundation
76+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-xctest
77+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-argument-parser
78+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-crypto
79+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-driver
80+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-llbuild llbuild
81+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-package-manager
82+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-system
83+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-tools-support-core
84+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-installer-scripts
85+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/indexstore-db
86+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/sourcekit-lsp
87+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/jpsim/Yams
88+
git clone -t curl-7_77_0 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/curl/curl
89+
git clone -t v2.9.12 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/gnome/libxml2
90+
git clone -t v1.2.11 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/madler/zlib
91+
git clone -b maint/maint-69 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/unicode-org/icu
92+
```
93+
94+
#### Run Docker
95+
96+
```cmd
97+
docker run --rm -it -v %UserProfile%\data:S: swift:swiftci
98+
```
99+
100+
#### Build the Toolchain
101+
102+
This will build a full Swift toolchain distribution (llvm, clang, lld, lldb,
103+
swift, swift-package-manger, SourceKit-LSP) and the Windows SDK (x86, x64,
104+
ARM64).
105+
106+
```cmd
107+
S:
108+
S:\SourceCache\swift\utils\build.cmd
109+
```
110+
111+
#### Running Swift Tests
112+
113+
The toolchain tests require some modifications to the path to find some of the
114+
dependencies. The following will run the Swift test suite within the docker
115+
container:
116+
117+
```cmd
118+
path S:\b\1\bin;S:\b\1\tools\swift\libdispatch-windows-x86_64-prefix\bin;%Path%;%ProgramFiles%\Git\usr\bin
119+
ninja -C S:\b\1 check-swift
120+
```
121+
122+
#### Using the Toolchain
123+
124+
> **NOTE**: Running the test suite and using the toolchain near the production mode are mututally incompatible (due to the path changes).
125+
126+
The build will generate a toolchain image that is roughly similar to the
127+
installed version. The following can be run inside the docker container to use
128+
the toolchain:
129+
130+
```cmd
131+
set SDKROOT=S:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk
132+
path S:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\bin\x64;S:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin;%Path%
133+
```
134+
135+
Because the toolchain is built in the volume which is backed by the host, the
136+
toolchain can be used on the host (assuming the dependencies such as Visual
137+
Studio is installed and the module modules deployed). The adjusted paths below
138+
should enable that:
139+
140+
```cmd
141+
set SDKROOT=%UserProfile%\data\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk
142+
path %UserProfile%\data\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\bin\x64;%UserProfile%\data\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin;%Path%
143+
```
144+
38145
## Contributions
39146

40147
Contributions via pull requests are welcome and encouraged :)

‎swift-ci/master/windows/10.0.19042.1645/Dockerfile

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# escape=`
2+
3+
FROM mcr.microsoft.com/windows/servercore:10.0.19042.1645 AS windows
4+
5+
LABEL maintainer="Swift Infrastructure <[email protected]>"
6+
LABEL description="Docker Container for the Swift programming language"
7+
8+
ARG GIT=https://github.com/git-for-windows/git/releases/download/v2.36.0.windows.1/Git-2.36.0-64-bit.exe
9+
ARG PYTHON=https://www.python.org/ftp/python/3.10.4/python-3.10.4-amd64.exe
10+
11+
# restore the default Windows shell for correct batch processing
12+
SHELL ["cmd", "/S", "/C"]
13+
14+
# Enable Developer Mode.
15+
RUN reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" /t REG_DWORD /f /v "AllowDevelopmentWithoutDevLicense" /d "1"
16+
17+
# Enable Long Paths
18+
RUN reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /t REG_DWORD /f /v "LongPathsEnabled" /d "1"
19+
20+
# Install Git.
21+
# See: git-[version]-[bit].exe /SAVEINF=git.inf and /?
22+
COPY git.inf .
23+
RUN `
24+
curl -SLo git.exe %GIT% `
25+
&& (start /w git.exe /SP- /VERYSILENT /SUPPRESSMSGBOXES /NOCANCEL /NORESTART /CLOSEAPPLICATIONS /FORCECLOSEAPPLICATIONS /LOADINF=git.inf ) `
26+
&& del /q git.exe git.inf
27+
28+
# Install Python.
29+
# See: https://docs.python.org/3.10/using/windows.html
30+
# FIXME: it appears that `PYTHONHOME` and `PYTHONPATH` are unset
31+
COPY unattend.xml .
32+
RUN `
33+
curl -SLo python.exe %PYTHON% `
34+
&& (start /w python.exe /quiet ) `
35+
&& del /q python.exe unattend.xml
36+
37+
# Install Visual Studio Build Tools
38+
RUN `
39+
curl -SLo vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe `
40+
&& (start /w vs_buildtools.exe --quiet --wait --norestart --nocache `
41+
--add Microsoft.VisualStudio.Component.Windows11SDK.22000 `
42+
--add Microsoft.VisualStudio.Component.VC.ATL `
43+
--add Microsoft.VisualStudio.Component.VC.ATL.ARM `
44+
--add Microsoft.VisualStudio.Component.VC.ATL.ARM64 `
45+
--add Microsoft.VisualStudio.Component.VC.CMake.Project `
46+
--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
47+
--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 `
48+
--add Microsoft.VisualStudio.Component.VC.Tools.ARM `
49+
|| IF "%EXITCODE%"=="3010" EXIT 0) `
50+
&& del /q vs_buildtools.exe
51+
COPY InstallModules.cmd .
52+
RUN InstallModules.cmd && del /q InstallModules.cmd
53+
54+
# FIXME: we should use a non-Administrator user
55+
# USER ContainerUser
56+
57+
ENV PYTHONUTF8=1
58+
# Default to powershell
59+
CMD ["powershell.exe", "-nologo", "-ExecutionPolicy", "Bypass"]

‎swift-ci/master/windows/10.0.19042.1645/InstallModules.cmd

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
@echo off
2+
setlocal
3+
set vswhere=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe
4+
FOR /F "tokens=* usebackq" %%r IN (`"%vswhere%" -nologo -latest -all -prerelease -products * -property installationPath`) DO SET VsDevCmd=%%r\Common7\Tools\VsDevCmd.bat
5+
CALL "%VsDevCmd%" -no_logo -host_arch=amd64 -arch=amd64
6+
mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\ucrt\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\ucrt.modulemap
7+
mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\um\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\winsdk.modulemap
8+
mklink "%VCToolsInstallDir%\include\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\visualc.modulemap
9+
mklink "%VCToolsInstallDir%\include\visualc.apinotes" S:\SourceCache\swift\stdlib\public\Platform\visualc.apinotes
10+
endlocal
11+
@echo on

‎swift-ci/master/windows/10.0.19042.1645/git.inf

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[Setup]
2+
NoIcons=1
3+
Components=gitlfs
4+
EditorOption=VIM
5+
PathOption=Cmd
6+
SSHOption=OpenSSH
7+
TurtoiseOption=false
8+
CURLOption=WinSSL
9+
BashTerminalOption=ConHost
10+
PerformanceTweaksFSCache=Enabled
11+
EnableSymlinks=Enabled
12+
EnablePseudoConsoltSupport=Disabled
13+
EnableFSMonitor=Enabled

‎swift-ci/master/windows/10.0.19042.1645/unattend.xml

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Options>
3+
<Option Name="InstallAllUsers" Value="1" />
4+
<Option Name="AssociateFiles" Value="0" />
5+
<Option Name="PrependPath" Value="1" />
6+
<Option Name="Shortcuts" Value="0" />
7+
<Option Name="Include_doc" Value="0" />
8+
<Option Name="Include_debug" Value="0" />
9+
<Option Name="Include_dev" Value="1" />
10+
<Option Name="Include_exe" Value="1" />
11+
<Option Name="Include_launcher" Value="0" />
12+
<Option Name="InstallLauncherAllUsers" Value="0" />
13+
<Option Name="Include_lib" Value="1" />
14+
<Option Name="Include_symbols" Value="0" />
15+
<Option Name="Include_tcltk" Value="0" />
16+
<Option Name="Include_test" Value="0" />
17+
<Option Name="Include_tools" Value="0" />
18+
</Options>

0 commit comments

Comments
 (0)