Fix test output during CI. (#652)

When tests run in CI, we are not displaying the test output to the console. So if a test fails, and for some reason the .trx file isn't parsed correctly, it is impossible to see what test failed and why.

The test output isn't being displayed because of microsoft/vstest#1503.

To work around the vstest bug, split CI builds into 2 separate MSBuild invocations: one to do the build (which is multi-proc) and another to run the tests (which doesn't need MSBuild node reuse).

Author: eerhardt

build.proj

@@ -34,7 +34,6 @@
       BuildNative;
       $(TraversalBuildDependsOn);
       DownloadExternalTestFiles;
-      RunTests;
     </TraversalBuildDependsOn>
   </PropertyGroup>
 
@@ -80,7 +79,7 @@
                          TreatErrorsAsWarnings="true"/>
   </Target>
 
-  <Target Name="RunTests" Condition="'$(RunTests)'=='true'">
+  <Target Name="RunTests">
     <MSBuild Projects="test\run-tests.proj"
              Targets="RunTests" />
   </Target>

build/ci/phase-template.yml

@@ -21,8 +21,10 @@ phases:
           _configuration: Release
       ${{ insert }}: ${{ parameters.queue }}
     steps:
+    - script: $(_buildScript) -$(_configuration)
+      displayName: Build
     - script: $(_buildScript) -$(_configuration) -runtests
-      displayName: Build and Test
+      displayName: Run Tests
     - task: PublishTestResults@2
       displayName: Publish Test Results
       condition: succeededOrFailed()

config.json

@@ -67,8 +67,8 @@
       "defaultValue": ""
     },
     "RunTests": {
-      "description": "Run tests after building.",
-      "valueType": "property",
+      "description": "MsBuild target that run the tests. Call this after building.",
+      "valueType": "target",
       "values": [],
       "defaultValue": ""
     },
@@ -113,9 +113,9 @@
           }
         },
         "runtests": {
-          "description": "Runs the tests after building.",
+          "description": "Runs the tests. Call this after building.",
           "settings": {
-            "RunTests": "true"
+            "RunTests": "default"
           }
         },
         "verbose": {

docs/building/windows-instructions.md

@@ -46,14 +46,14 @@ You can use the Developer Command Prompt, Powershell or work in any regular cmd.
 From a (non-admin) Command Prompt window:
 
 - `build.cmd` - builds the assemblies
-- `build.cmd -runTests` - builds the assemblies and runs tests
+- `build.cmd -runTests` - called after a normal "build.cmd" will run all tests
 - `build.cmd -buildPackages` called after a normal “build.cmd” will create the NuGet packages with the assemblies in “bin"
 
 **Note**: Before working on individual projects or test projects you **must** run `build.cmd` from the root once before beginning that work. It is also a good idea to run `build.cmd` whenever you pull a large set of unknown changes into your branch.
 
 ### Running tests from the command line
 
-From the root, use `build.cmd -runTests`.
+From the root, run `build.cmd` and then `build.cmd -runTests`.
 For more details, or to test an individual project, you can navigate to the test project directory and then use `dotnet test`
 
 ### Running tests from Visual Studio

docs/project-docs/developer-guide.md

@@ -32,7 +32,8 @@ build.cmd -Release -TargetArchitecture:x64
 
 - Building the src and then building and running the tests
 ```
-build.cmd -RunTests
+build.cmd
+build.cmd -runTests
 ```
 
 ### Building individual projects

netci.groovy

@@ -16,6 +16,7 @@ def branch = GithubBranchName
 
             def newJob = job(Utilities.getFullJobName(project, jobName, isPR)) {
                 steps {
+                    shell("./build.sh -$config")
                     shell("./build.sh -$config -runtests")
                     shell("./build.sh -buildPackages")
                 }