Skip to content

Commit 7f9136a

Browse files
benbpbenbp
authored
Copy job matrix functionality (#16450)
1 parent 8d50208 commit 7f9136a

7 files changed

+1480
-0
lines changed

‎eng/scripts/job-matrix/Create-JobMatrix.ps1

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
<#
2+
.SYNOPSIS
3+
Generates a JSON object representing an Azure Pipelines Job Matrix.
4+
See https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#parallelexec
5+
6+
.EXAMPLE
7+
.\eng\scripts\Create-JobMatrix $context
8+
#>
9+
10+
[CmdletBinding()]
11+
param (
12+
[Parameter(Mandatory=$True)][string] $ConfigPath,
13+
[Parameter(Mandatory=$True)][string] $Selection,
14+
[Parameter(Mandatory=$False)][string] $DisplayNameFilter,
15+
[Parameter(Mandatory=$False)][array] $Filters
16+
)
17+
18+
. $PSScriptRoot/job-matrix-functions.ps1
19+
20+
$config = GetMatrixConfigFromJson (Get-Content $ConfigPath)
21+
22+
[array]$matrix = GenerateMatrix $config $Selection $DisplayNameFilter $Filters
23+
$serialized = SerializePipelineMatrix $matrix
24+
25+
Write-Output $serialized.pretty
26+
Write-Output "##vso[task.setVariable variable=matrix;isOutput=true]$($serialized.compressed)"

‎eng/scripts/job-matrix/README.md

+274
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
# Azure Pipelines Matrix Generator
2+
3+
* [Usage in a pipeline](#usage-in-a-pipeline)
4+
* [Matrix config file syntax](#matrix-config-file-syntax)
5+
* [Fields](#fields)
6+
* [matrix](#matrix)
7+
* [include](#include)
8+
* [exclude](#exclude)
9+
* [displayNames](#displaynames)
10+
* [Matrix Generation behavior](#matrix-generation-behavior)
11+
* [all](#all)
12+
* [sparse](#sparse)
13+
* [include/exclude](#includeexclude)
14+
* [displayNames](#displaynames-1)
15+
* [Filters](#filters)
16+
* [Under the hood](#under-the-hood)
17+
* [Testing](#testing)
18+
19+
20+
This directory contains scripts supporting dynamic, cross-product matrix generation for azure pipeline jobs.
21+
It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
22+
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and programmable matrix filters.
23+
24+
This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
25+
for a job matrix definition](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
26+
27+
## Usage in a pipeline
28+
29+
In order to use these scripts in a pipeline, you must provide a config file and call the matrix creation script within a powershell job.
30+
31+
For a single matrix, you can include the `eng/pipelines/templates/jobs/job-matrix.yml` template in a pipeline:
32+
33+
```
34+
jobs:
35+
- template: /eng/pipelines/templates/jobs/job-matrix.yml
36+
parameters:
37+
MatrixConfigs:
38+
- Name: base_product_matrix
39+
Path: /eng/pipelines/matrix.json
40+
Selection: sparse
41+
GenerateVMJobs: true
42+
- Name: sdk_specific_matrix
43+
Path: /sdk/foobar/matrix.json
44+
Selection: all
45+
GenerateContainerJobs: true
46+
steps:
47+
- pwsh:
48+
...
49+
```
50+
51+
## Matrix config file syntax
52+
53+
Matrix parameters can either be a list of strings, or a set of grouped strings (represented as a hash). The latter parameter
54+
type is useful for when 2 or more parameters need to be grouped together, but without generating more than one matrix permutation.
55+
56+
```
57+
"matrix": {
58+
"<parameter1 name>": [ <values...> ],
59+
"<parameter2 name>": [ <values...> ],
60+
"<parameter set>": {
61+
"<parameter set 1 name>": {
62+
"<parameter set 1 value 1": "value",
63+
"<parameter set 1 value 2": "<value>",
64+
},
65+
"<parameter set 2 name>": {
66+
"<parameter set 2 value 1": "value",
67+
"<parameter set 2 value 2": "<value>",
68+
}
69+
}
70+
}
71+
"include": [ <matrix>, <matrix>, ... ],
72+
"exclude": [ <matrix>, <matrix>, ... ],
73+
"displayNames": { <parameter value>: <human readable override> }
74+
```
75+
76+
See `samples/matrix.json` for a full sample.
77+
78+
### Fields
79+
80+
#### matrix
81+
82+
The `matrix` field defines the base cross-product matrix. The generated matrix can be full or sparse.
83+
84+
Example:
85+
```
86+
"matrix": {
87+
"operatingSystem": [
88+
"windows-2019",
89+
"ubuntu-18.04",
90+
"macOS-10.15"
91+
],
92+
"framework": [
93+
"net461",
94+
"netcoreapp2.1",
95+
"net50"
96+
],
97+
"additionalTestArguments": [
98+
"",
99+
"/p:UseProjectReferenceToAzureClients=true",
100+
]
101+
}
102+
```
103+
104+
#### include
105+
106+
The `include` field defines any number of matrices to be appended to the base matrix after processing exclusions.
107+
108+
#### exclude
109+
110+
The `include` field defines any number of matrices to be removed from the base matrix. Exclude parameters can be a partial
111+
set, meaning as long as all exclude parameters match against a matrix entry (even if the matrix entry has additional parameters),
112+
then it will be excluded from the matrix. For example, the below entry will match the exclusion and be removed:
113+
114+
```
115+
matrix entry:
116+
{
117+
"a": 1,
118+
"b": 2,
119+
"c": 3,
120+
}
121+
122+
"exclude": [
123+
{
124+
"a": 1,
125+
"b": 2
126+
}
127+
]
128+
```
129+
130+
#### displayNames
131+
132+
Specify any overrides for the azure pipelines definition and UI that determines the matrix job name. If some parameter
133+
values are too long or unreadable for this purpose (e.g. a command line argument), then you can replace them with a more
134+
readable value here. For example:
135+
136+
```
137+
"displayNames": {
138+
"/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
139+
},
140+
"matrix": {
141+
"additionalTestArguments": [
142+
"/p:UseProjectReferenceToAzureClients=true"
143+
]
144+
}
145+
```
146+
147+
## Matrix Generation behavior
148+
149+
#### all
150+
151+
`all` will output the full matrix, i.e. every possible permutation of all parameters given (p1.Length * p2.Length * ...).
152+
153+
#### sparse
154+
155+
`sparse` outputs the minimum number of parameter combinations while ensuring that all parameter values are present in at least one matrix job.
156+
Effectively this means the total length of a sparse matrix will be equal to the largest matrix dimension, i.e. `max(p1.Length, p2.Length, ...)`.
157+
158+
To build a sparse matrix, a full matrix is generated, and then walked diagonally N times where N is the largest matrix dimension.
159+
This pattern works for any N-dimensional matrix, via an incrementing index (n, n, n, ...), (n+1, n+1, n+1, ...), etc.
160+
Index lookups against matrix dimensions are calculated modulus the dimension size, so a two-dimensional matrix of 4x2 might be walked like this:
161+
162+
```
163+
index: 0, 0:
164+
o . . .
165+
. . . .
166+
167+
index: 1, 1:
168+
. . . .
169+
. o . .
170+
171+
index: 2, 2 (modded to 2, 0):
172+
. . o .
173+
. . . .
174+
175+
index: 3, 3 (modded to 3, 1):
176+
. . . .
177+
. . . o
178+
```
179+
180+
#### include/exclude
181+
182+
Include and exclude support additions and subtractions off the base matrix. Both include and exclude take an array of matrix values.
183+
Typically these values will be a single entry, but they also support the cross-product matrix definition syntax of the base matrix.
184+
185+
Include and exclude are parsed fully. So if a sparse matrix is called for, a sparse version of the base matrix will be generated, but
186+
the full matrix of both include and exclude will be processed.
187+
188+
Excludes are processed first, so includes can be used to add back any specific jobs to the matrix.
189+
190+
#### displayNames
191+
192+
In the matrix job output that azure pipelines consumes, the format is a dictionary of dictionaries. For example:
193+
194+
```
195+
{
196+
"net461_macOS1015": {
197+
"framework": "net461",
198+
"operatingSystem": "macOS-10.15"
199+
},
200+
"net50_ubuntu1804": {
201+
"framework": "net50",
202+
"operatingSystem": "ubuntu-18.04"
203+
},
204+
"netcoreapp21_windows2019": {
205+
"framework": "netcoreapp2.1",
206+
"operatingSystem": "windows-2019"
207+
},
208+
"UseProjectRef_net461_windows2019": {
209+
"additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true",
210+
"framework": "net461",
211+
"operatingSystem": "windows-2019"
212+
}
213+
}
214+
```
215+
216+
The top level keys are used as job names, meaning they get displayed in the azure pipelines UI when running the pipeline.
217+
218+
The logic for generating display names works like this:
219+
220+
- Join parameter values by "_"
221+
a. If the parameter value exists as a key in `displayNames` in the matrix config, replace it with that value.
222+
b. For each name value, strip all non-alphanumeric characters (excluding "_").
223+
c. If the name is greater than 100 characters, truncate it.
224+
225+
#### Filters
226+
227+
Filters can be passed to the matrix as an array of strings, each matching the format of <key>=<regex>. When a matrix entry
228+
does not contain the specified key, it will default to a value of empty string for regex parsing. This can be used to specify
229+
filters for keys that don't exist or keys that optionally exist and match a regex, as seen in the below example.
230+
231+
Display name filters can also be passed as a single regex string that runs against the [generated display name](#displaynames) of the matrix job.
232+
The intent of display name filters is to be defined primarily as a top level variable at template queue time in the azure pipelines UI.
233+
234+
For example, the below command will filter for matrix entries with "windows" in the job display name, no matrix variable
235+
named "ExcludedKey", a framework variable containing either "461" or "5.0", and an optional key "SupportedClouds" that, if exists, must contain "Public":
236+
237+
```
238+
./Create-JobMatrix.ps1 `
239+
-ConfigPath samples/matrix.json `
240+
-Selection all `
241+
-DisplayNameFilter ".*windows.*" `
242+
-Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
243+
```
244+
245+
#### Under the hood
246+
247+
The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
248+
the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
249+
250+
```
251+
"matrix": {
252+
"framework": [ "net461", "netcoreapp2.1" ],
253+
"additionalTestArguments": [ "", "/p:SuperTest=true" ]
254+
"pool": [ "ubuntu-18.04" ],
255+
"container": [ "ubuntu-18.04" ],
256+
"testMode": [ "Record" ]
257+
}
258+
```
259+
260+
The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
261+
262+
## Testing
263+
264+
The matrix functions can be tested using [pester](https://pester.dev/):
265+
266+
```
267+
$ Invoke-Pester
268+
269+
Starting discovery in 1 files.
270+
Discovery finished in 384ms.
271+
[+] /home/ben/sdk/azure-sdk-for-net/eng/scripts/job-matrix/job-matrix-functions.tests.ps1 4.09s (1.52s|2.22s)
272+
Tests completed in 4.12s
273+
Tests Passed: 120, Failed: 0, Skipped: 4 NotRun: 0
274+
```

‎eng/scripts/job-matrix/job-matrix-functions.filter.tests.ps1

+84
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
Import-Module Pester
2+
3+
BeforeAll {
4+
. ./job-matrix-functions.ps1
5+
6+
$matrixConfig = @"
7+
{
8+
"matrix": {
9+
"operatingSystem": [ "windows-2019", "ubuntu-18.04", "macOS-10.15" ],
10+
"framework": [ "net461", "netcoreapp2.1" ],
11+
"additionalArguments": [ "", "mode=test" ]
12+
}
13+
}
14+
"@
15+
$config = GetMatrixConfigFromJson $matrixConfig
16+
}
17+
18+
Describe "Matrix Filter" -Tag "filter" {
19+
It "Should filter by matrix display name" -TestCases @(
20+
@{ regex = "windows.*"; expectedFirst = "windows2019_net461"; length = 4 }
21+
@{ regex = "windows2019_netcoreapp21_modetest"; expectedFirst = "windows2019_netcoreapp21_modetest"; length = 1 }
22+
@{ regex = ".*ubuntu.*"; expectedFirst = "ubuntu1804_net461"; length = 4 }
23+
) {
24+
[array]$matrix = GenerateMatrix $config "all" $regex
25+
$matrix.Length | Should -Be $length
26+
$matrix[0].Name | Should -Be $expectedFirst
27+
}
28+
29+
It "Should handle no display name filter matches" {
30+
$matrix = GenerateMatrix $config "all"
31+
[array]$filtered = FilterMatrixDisplayName $matrix "doesnotexist"
32+
$filtered | Should -BeNullOrEmpty
33+
}
34+
35+
It "Should filter by matrix key/value" -TestCases @(
36+
@{ filterString = "operatingSystem=windows.*"; expectedFirst = "windows2019_net461"; length = 4 }
37+
@{ filterString = "operatingSystem=windows-2019"; expectedFirst = "windows2019_net461"; length = 4 }
38+
@{ filterString = "framework=.*"; expectedFirst = "windows2019_net461"; length = 12 }
39+
@{ filterString = "additionalArguments=mode=test"; expectedFirst = "windows2019_net461_modetest"; length = 6 }
40+
@{ filterString = "additionalArguments=^$"; expectedFirst = "windows2019_net461"; length = 6 }
41+
) {
42+
[array]$matrix = GenerateMatrix $config "all" -filters @($filterString)
43+
$matrix.Length | Should -Be $length
44+
$matrix[0].Name | Should -Be $expectedFirst
45+
}
46+
47+
It "Should filter by optional matrix key/value" -TestCases @(
48+
@{ filterString = "operatingSystem=^$|windows.*"; expectedFirst = "windows2019_net461"; length = 4 }
49+
@{ filterString = "doesnotexist=^$|.*"; expectedFirst = "windows2019_net461"; length = 12 }
50+
) {
51+
[array]$matrix = GenerateMatrix $config "all" -filters @($filterString)
52+
$matrix.Length | Should -Be $length
53+
$matrix[0].Name | Should -Be $expectedFirst
54+
}
55+
56+
It "Should handle multiple matrix key/value filters " {
57+
[array]$matrix = GenerateMatrix $config "all" -filters "operatingSystem=windows.*","framework=.*","additionalArguments=mode=test"
58+
$matrix.Length | Should -Be 2
59+
$matrix[0].Name | Should -Be "windows2019_net461_modetest"
60+
}
61+
62+
It "Should handle no matrix key/value filter matches" {
63+
[array]$matrix = GenerateMatrix $config "all" -filters @("doesnot=exist")
64+
$matrix | Should -BeNullOrEmpty
65+
}
66+
67+
It "Should handle invalid matrix key/value filter syntax" {
68+
{ GenerateMatrix $config "all" -filters @("invalid") } | Should -Throw
69+
{ GenerateMatrix $config "all" -filters @("emptyvalue=") } | Should -Throw
70+
{ GenerateMatrix $config "all" -filters @("=emptykey") } | Should -Throw
71+
{ GenerateMatrix $config "all" -filters @("=") } | Should -Throw
72+
}
73+
74+
It "Should filter by key exclude" {
75+
[array]$matrix = GenerateMatrix $config "all" -filters @("operatingSystem=^$")
76+
$matrix | Should -BeNullOrEmpty
77+
78+
[array]$matrix = GenerateMatrix $config "all"
79+
$matrix.Length | Should -Be 12
80+
$matrix += @{ Name = "excludeme"; Parameters = [Ordered]@{ "foo" = 1 } }
81+
[array]$matrix = FilterMatrix $matrix @("foo=^$")
82+
$matrix.Length | Should -Be 12
83+
}
84+
}

0 commit comments

Comments
 (0)