Skip to content

Commit 8bc5616

Browse files
sbomersbomer
authored
Add azure-pipelines build and test definitions (dotnet#20840)
This adds an azure pipeline definition with a matrix of product and test builds, using helix to run tests. The intention is that this definition will eventually be used for both our official build and CI testing. There is one build job for each OS/platform/arch, and one test job for each OS/platform/arch/priority/R2Rflag. The test job builds tests and then submits them to helix, passing along a number of test run modes. One helix test job will be created for each OS/platform/arch/priority/R2Rflag/helixtargetqueue/testscenario. There is a lot of work left to be done to get this up to parity with our official builds and CI, which I've tried to call out in comments.
1 parent f6e7568 commit 8bc5616

16 files changed

+846
-13
lines changed

‎NuGet.Config

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<configuration>
3+
<packageSources>
4+
<clear />
5+
<add key="arcade" value="https://dotnetfeed.blob.core.windows.net/dotnet-tools-internal/index.json" />
6+
<add key="dotnet-core" value="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json" />
7+
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
8+
</packageSources>
9+
</configuration>

‎azure-pipelines.yml

+178
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
resources:
2+
repositories:
3+
# shared library repository
4+
- repository: arcade
5+
type: github
6+
endpoint: DotNet-Bot GitHub Connection
7+
name: dotnet/arcade
8+
ref: refs/heads/master
9+
10+
variables:
11+
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
12+
13+
14+
jobs:
15+
16+
## The following is the matrix of test runs that we have. This is
17+
## duplicated for each os/arch combination in platform-matrix.yml.
18+
19+
##
20+
## Product build Test build Test run
21+
## (Azure DevOps) (Azure DevOps) (helix)
22+
##
23+
## ###########################################################################################
24+
##
25+
## Debug
26+
##
27+
## Checked ----------> Pri0 -----------------> plain runtests
28+
## |
29+
## \---------> Pri1 -----------------> plain runtests
30+
## | \----------------> jitstress
31+
## | \----------------> gcstress
32+
## | \----------------> maybe more (dynamically selected runtest modes)
33+
## |
34+
## \---------> Pri1 crossgen --------> plain runtests
35+
## \-------> jitstress
36+
## \-------> gcstress
37+
## \-------> maybe more (dynamically selected runtest modes)
38+
##
39+
## Release ----------> Pri1 -----------------> plain runtests
40+
## |
41+
## \---------> Pri1 crossgen --------> plain runtests
42+
##
43+
##
44+
45+
## Each build or test job is defined in Azure DevOps, and will show
46+
## up in the UI in the order in which they are defined here. The
47+
## build and test build job matrix is defined statically, but
48+
## queue-time inputs can be used to control whether a job executes
49+
## (used to select which jobs run in ci vs for official builds), or
50+
## to select test modes. This should eventually be used to enable
51+
## requesting specific test runs from pull requests.
52+
53+
54+
##
55+
## Templates used to define jobs:
56+
## Please update this if the factoring changes.
57+
##
58+
## This file defines the set of jobs in a platform-agnostic manner,
59+
## using the platform-matrix.yml template. This will create one job
60+
## for each platform from the passed-in jobTemplate (either a build
61+
## job or a test job). The build-job.yml and test-job.yml templates
62+
## use xplat-job.yml to handle some of the common logic for
63+
## abstracting over platforms. Finally, xplat-job.yml uses the arcade
64+
## base.yml job template, which sets up telemetry and signing support.
65+
66+
## azure-pipelines.yml -> platform-matrix.yml -------> build-job.yml -------> xplat-job.yml -> base.yml
67+
## | (passed-in jobTemplate) | (arcade)
68+
## \------> test-job.yml ------/
69+
70+
71+
72+
#
73+
# Debug build
74+
#
75+
76+
- template: eng/platform-matrix.yml
77+
parameters:
78+
jobTemplate: build-job.yml
79+
buildConfig: debug
80+
81+
#
82+
# Checked build
83+
#
84+
85+
- template: eng/platform-matrix.yml
86+
parameters:
87+
jobTemplate: build-job.yml
88+
buildConfig: checked
89+
90+
#
91+
# Release build
92+
#
93+
94+
- template: eng/platform-matrix.yml
95+
parameters:
96+
jobTemplate: build-job.yml
97+
buildConfig: release
98+
99+
#
100+
# Checked test builds
101+
#
102+
103+
# Pri0
104+
- template: eng/platform-matrix.yml
105+
parameters:
106+
jobTemplate: test-job.yml
107+
buildConfig: checked
108+
jobParameters:
109+
priority: 0
110+
111+
# Pri1
112+
- template: eng/platform-matrix.yml
113+
parameters:
114+
jobTemplate: test-job.yml
115+
buildConfig: checked
116+
jobParameters:
117+
priority: 1
118+
scenarios: 'normal;jitstress2'
119+
120+
# Pri1 crossgen
121+
- template: eng/platform-matrix.yml
122+
parameters:
123+
jobTemplate: test-job.yml
124+
buildConfig: checked
125+
jobParameters:
126+
priority: 1
127+
crossgen: true
128+
scenarios: 'normal;jitstress2'
129+
130+
#
131+
# Release test builds
132+
#
133+
134+
# Pri1
135+
- template: eng/platform-matrix.yml
136+
parameters:
137+
jobTemplate: test-job.yml
138+
buildConfig: release
139+
jobParameters:
140+
priority: 1
141+
142+
# Pri1 crossgen
143+
- template: eng/platform-matrix.yml
144+
parameters:
145+
jobTemplate: test-job.yml
146+
buildConfig: release
147+
jobParameters:
148+
priority: 1
149+
crossgen: true
150+
151+
152+
# Publish build information to Build Assets Registry
153+
154+
# This job gathers build assets from the pipeline (from each official
155+
# product build job), and publishes them to the build assets
156+
# registry. Its dependencies should be updated to include all of the
157+
# official builds if we add more platform/arch combinations.
158+
159+
# TODO: Enable publish to BAR
160+
#- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
161+
# - template: /eng/common/templates/phases/publish-build-assets.yml@arcade
162+
#- phase: publish_bar
163+
# displayName: publish to BAR (empty currently)
164+
# queue:
165+
# name: Hosted VS2017
166+
# dependsOn:
167+
# - build_Linux_x64_release
168+
# - build_OSX_x64_release
169+
# - build_Windows_NT_x64_release
170+
# - build_Windows_NT_x86_release
171+
# - build_Windows_NT_arm_release
172+
# - build_Windows_NT_arm64_release
173+
# TODO: enable these builds
174+
# - build_rhel_x64_release
175+
# - build_alpine_x64_release
176+
# - build_crossbuild_arm_release
177+
# - build_crossbuild_arm64_release
178+

‎build-test.cmd

+1-1
Original file line numberDiff line numberDiff line change
@@ -527,7 +527,7 @@ set __MsbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
527527
set __MsbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
528528
set __Logging=-MsBuildLog=!__MsbuildLog! -MsBuildWrn=!__MsbuildWrn! -MsBuildErr=!__MsbuildErr!
529529

530-
call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\helixprep.proj !__Logging! %__RunArgs% %__BuildAgainstPackagesArg% %RuntimeIdArg% %TargetsWindowsArg% %__CrossgenArg% %__PriorityArg% %__PassThroughArg% %__UnprocessedBuildArgs%
530+
call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\helixprep.proj !__Logging! %__RunArgs% %RuntimeIdArg% %TargetsWindowsArg% %__CrossgenArg% %__PriorityArg% %__PassThroughArg% %__UnprocessedBuildArgs%
531531
if errorlevel 1 (
532532
echo %__MsgPrefix%Error: build failed. Refer to the build log files for details:
533533
echo %__BuildLog%

‎dir.common.props

+3-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,9 @@
2222
<BuildType Condition="'$(__BuildType)' == 'checked'">Checked</BuildType>
2323

2424
<BuildOS>$(__BuildOS)</BuildOS>
25-
<BuildOS Condition="'$(__BuildOS)' == ''">Windows_NT</BuildOS>
25+
<BuildOS Condition="'$(__BuildOS)' == '' And '$([MSBuild]::IsOSPlatform(Windows))' == 'true'">Windows_NT</BuildOS>
26+
<BuildOS Condition="'$(__BuildOS)' == '' And '$([MSBuild]::IsOSPlatform(Linux))' == 'true'">Linux</BuildOS>
27+
<BuildOS Condition="'$(__BuildOS)' == '' And '$([MSBuild]::IsOSPlatform(OSX))' == 'true'">OSX</BuildOS>
2628

2729
<Configuration Condition="'$(Configuration)' == ''">$(BuildType)</Configuration>
2830
<Platform Condition="'$(Platform)' == ''">$(BuildArch)</Platform>

‎eng/build-job.yml

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
parameters:
2+
buildConfig: ''
3+
archType: ''
4+
osGroup: ''
5+
6+
### Product build
7+
jobs:
8+
- template: xplat-job.yml
9+
parameters:
10+
buildConfig: ${{ parameters.buildConfig }}
11+
archType: ${{ parameters.archType }}
12+
osGroup: ${{ parameters.osGroup }}
13+
14+
# Compute job name from template parameters
15+
name: ${{ format('build_{0}_{1}_{2}', parameters.osGroup, parameters.archType, parameters.buildConfig) }}
16+
displayName: ${{ format('Build {0} {1} {2}', parameters.osGroup, parameters.archType, parameters.buildConfig) }}
17+
18+
steps:
19+
20+
# Install native dependencies
21+
- ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
22+
- script: sh eng/install-native-dependencies.sh $(osGroup)
23+
displayName: Install native dependencies
24+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
25+
# Necessary to install python
26+
- script: eng\common\init-tools-native.cmd -InstallDirectory $(Build.SourcesDirectory)\native-tools -Force
27+
displayName: Install native dependencies
28+
29+
30+
# Run init-tools (pre-arcade dependency bootstrapping)
31+
# TODO: replace this with an arcade equivalent
32+
- ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
33+
- script: ./init-tools.sh
34+
displayName: Init tools
35+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
36+
- script: .\init-tools.cmd
37+
displayName: Init tools
38+
39+
40+
# Sync
41+
- ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
42+
- script: ./Tools/dotnetcli/dotnet msbuild build.proj /p:RestoreDuringBuild=true /t:Sync
43+
displayName: Sync
44+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
45+
- script: .\Tools\dotnetcli\dotnet.exe msbuild build.proj /p:RestoreDuringBuild=true /t:Sync
46+
displayName: Sync
47+
48+
49+
# Build
50+
- ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
51+
- script: ./build.sh $(buildConfig) $(archType) -skipnuget -skiprestore
52+
displayName: Build product
53+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
54+
# TODO: IBCOptimize? EnforcePGO? pass an OfficialBuildId? SignType? file logging parameters?
55+
- script: set __TestIntermediateDir=int&&build.cmd $(buildConfig) $(archType) -skiptests -skipbuildpackages -skiprestore
56+
displayName: Build product
57+
58+
59+
# Upload build as pipeline artifact
60+
- ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
61+
- task: PublishPipelineArtifact@0
62+
displayName: Save product build as pipeline artifact
63+
inputs:
64+
artifactName: ${{ format('{0}_{1}_{2}_build', parameters.osGroup, parameters.archType, parameters.buildConfig) }}
65+
targetPath: $(Build.SourcesDirectory)/bin/Product/$(osGroup).$(archType).$(buildConfigUpper)
66+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
67+
- task: PublishPipelineArtifact@0
68+
displayName: Save product build as pipeline artifact
69+
inputs:
70+
artifactName: ${{ format('{0}_{1}_{2}_build', parameters.osGroup, parameters.archType, parameters.buildConfig) }}
71+
targetPath: $(Build.SourcesDirectory)\bin\Product\Windows_NT.$(archType).$(buildConfigUpper)
72+
73+
74+
# TODO: Sign
75+
- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
76+
- script: echo Sign!
77+
displayName: Sign Binaries (empty for now)
78+
79+
80+
# Get key vault secrets for publishing
81+
- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
82+
- task: AzureKeyVault@1
83+
inputs:
84+
azureSubscription: 'DotNet-Engineering-Services_KeyVault'
85+
KeyVaultName: EngKeyVault
86+
SecretsFilter: 'dotnetfeed-storage-access-key-1,microsoft-symbol-server-pat,symweb-symbol-server-pat'
87+
88+
89+
# TODO: Build packages and publish official build
90+
#- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
91+
# ${{ if or(eq(parameters.osGroup, 'Linux'), eq(parameters.osGroup, 'OSX')) }}:
92+
# # TODO: ensure that NUGET_PACKAGES path is correctly set
93+
# - script: .dotnet/dotnet msbuild src/restorearcadepublishtasks.proj /t:Restore
94+
# displayName: Restore tasks used to publish the build
95+
# - script: .dotnet/dotnet msbuild src/publishwitharcade.proj /t:PublishPackages /p:AzureFeedUrl=$(AzureFeedUrl) /p:AccountKey=$(AccountKey)
96+
# displayName: Publish official build
97+
# ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
98+
# # TODO: pass publish feed url and access token in from the internal pipeline
99+
# - script: .dotnet\dotnet.exe msbuild src\restorearcadepublishtasks.proj /t:Restore
100+
# displayName: Restore tasks used to publish the build
101+
# - script: .dotnet\dotnet.exe msbuild src\publishwitharcade.proj /t:PublishPackages /p:AzureFeedUrl=$(AzureFeedUrl) /p:AccountKey=$(AccountKey)
102+
103+
# Kill tasks that hold onto files on Windows. Otherwise git clean
104+
# may fail for later jobs on the same agent.
105+
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
106+
- script: eng/kill_tasks.cmd
107+
displayName: Kill tasks that hold on to files
108+
condition: always()

‎eng/install-native-dependencies.sh

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/usr/bin/env sh
2+
3+
if [ "$1" = "Linux" ]; then
4+
sudo apt update
5+
if [ "$?" != "0" ]; then
6+
exit 1;
7+
fi
8+
sudo apt install cmake llvm-3.9 clang-3.9 lldb-3.9 liblldb-3.9-dev libunwind8 libunwind8-dev gettext libicu-dev liblttng-ust-dev libcurl4-openssl-dev libssl-dev libkrb5-dev libnuma-dev
9+
if [ "$?" != "0"]; then
10+
exit 1;
11+
fi
12+
elif [ "$1" = "OSX" ]; then
13+
brew install icu4c openssl
14+
if [ "$?" != "0" ]; then
15+
exit 1;
16+
fi
17+
brew link --force icu4c
18+
if [ "$?" != "0"]; then
19+
exit 1;
20+
fi
21+
else
22+
echo "Must pass \"Linux\" or \"OSX\" as first argument."
23+
exit 1
24+
fi
25+

‎eng/kill_tasks.cmd

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
@if not defined _echo @echo off
2+
setlocal EnableDelayedExpansion
3+
4+
:: Check if VBCSCompiler.exe is running
5+
tasklist /fi "imagename eq VBCSCompiler.exe" |find ":" > nul
6+
:: Compiler is running if errorlevel == 1
7+
if errorlevel 1 (
8+
echo Stop VBCSCompiler.exe execution.
9+
for /f "tokens=2 delims=," %%F in ('tasklist /nh /fi "imagename eq VBCSCompiler.exe" /fo csv') do taskkill /f /PID %%~F
10+
)

0 commit comments

Comments
 (0)