Threshold value not working

[jacogreyling]

Hi,

I just upgraded to the latest v1.1.0 GA build. I'm using the new /p:Threshold=80 option to the build process yet the test run still returns 'successful' even if the coverage is below the threshold.

This is the command I run to build:

dotnet test --configuration Release --logger 'trx;logfilename=mstest-results.trx' /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutputDirectory=TestResults /p:Threshold=80

This is the output:

Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:01.5821864]   Discovering: UnitTests
[xUnit.net 00:00:01.7179983]   Discovered:  UnitTests
[xUnit.net 00:00:01.7245869]   Starting:    UnitTests
[xUnit.net 00:00:01.9276830]     AuthResponsesOperationFilter does not add 401 response to operation without AuthoriseAttribute [SKIP]
[xUnit.net 00:00:01.9282262]       Cannot yet mock ApiDescription objects
Skipped  AuthResponsesOperationFilter does not add 401 response to operation without AuthoriseAttribute
Standard Output Messages:
 Cannot yet mock ApiDescription objects
[xUnit.net 00:00:01.9862873]     AuthResponsesOperationFilter Adds 401 response to operation with AuthoriseAttribute [SKIP]
[xUnit.net 00:00:01.9863816]       Cannot yet mock ApiDescription objects
Skipped  AuthResponsesOperationFilter Adds 401 response to operation with AuthoriseAttribute
Standard Output Messages:
 Cannot yet mock ApiDescription objects
[xUnit.net 00:00:02.3872529]   Finished:    UnitTests

Total tests: 33. Passed: 31. Failed: 0. Skipped: 2.
Test Run Successful.
Test execution time: 3.8409 Seconds
WARNING: Overwriting results file: /home/jenkins/agent/workspace/an-updated-coverage-process-7N7FVRFUDPWSNYEY7B7OE6IM5VJ26KCY5NFVLTNHW4HURJ65KZAQ/tests/UnitTests/TestResults/mstest-results.trx
Results File: /home/jenkins/agent/workspace/an-updated-coverage-process-7N7FVRFUDPWSNYEY7B7OE6IM5VJ26KCY5NFVLTNHW4HURJ65KZAQ/tests/UnitTests/TestResults/mstest-results.trx

Calculating coverage result...
  Generating report 'TestResults/coverage.xml'

| Module                 | Coverage |
|------------------------|----------|
| ApplicationCore        | 75%      |
| Infrastructure         | 37.8%    |
| Nedbank.Authentication | 0%       |
| Web                    | 42.9%    |

Am I missing something or is this not working right now.

Also my build environment is:

Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
We're running .NET Core 2.0 on Linux. (Install .NET Core SDK 2.1.104 Package)

[tonerdo]

The Threshold value does work, if you check the last exit code you'll see it's a non zero value. Since the coverage task is the last, MSBuild doesn't issue an error. If it wasn't, MSBuild won't proceed to other targets. To make it more obvious I'll log an error to the output

[tonerdo]

New NuGet bits: https://www.nuget.org/packages/coverlet.msbuild/1.1.1

[jacogreyling]

Thank you very much @tonerso! An error log message is exactly what I need. This will make the reason more obvious in Jenkins why it failed (vs any other build error).

[linxingquan]

hi @tonerdo I still have the issue with the threshold. The tests succeed without any error. I'm using coverlet.msbuild 3.2.0.

PS D:\src\DataExtraction\src\DataExtractionDotnet\BinaryAnalyzerCIL.Tests> dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:Threshold=90
  Determining projects to restore...
  All projects are up-to-date for restore.
  DataExtractionCommon -> D:\src\DataExtraction\src\DataExtractionDotnet\DataExtractionCommon\bin\Debug\net6.0\DataExtractionComm
  on.dll
  DataExtractionCommon.Tests -> D:\src\DataExtraction\src\DataExtractionDotnet\DataExtractionCommon.Tests\bin\Debug\net6.0\DataEx
  tractionCommon.Tests.dll
  BinaryAnalyzerCIL -> D:\src\DataExtraction\src\DataExtractionDotnet\BinaryAnalyzerCIL\bin\Debug\net6.0\BinaryAnalyzerCIL.dll
  DataExtraction.GraphAnnotation -> D:\src\DataExtraction\src\DataExtractionDotnet\DataExtraction.GraphAnnotation\bin\Debug\net6.
  0\DataExtraction.GraphAnnotation.dll
  DataExtractorCIL -> D:\src\DataExtraction\src\DataExtractionDotnet\DataExtractorCIL\bin\Debug\net6.0\DataExtract
  orCIL.dll
  BinaryAnalyzerCIL.Tests -> D:\src\DataExtraction\src\DataExtractionDotnet\BinaryAnalyzerCIL.Tests\bin\Debug\net6.0\BinaryAnalyz
  erCIL.Tests.dll
Test run for D:\src\DataExtraction\src\DataExtractionDotnet\BinaryAnalyzerCIL.Tests\bin\Debug\net6.0\BinaryAnalyzerCIL.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:    97, Skipped:     0, Total:    97, Duration: 9 s - BinaryAnalyzerCIL.Tests.dll (net6.0)

Calculating coverage result...
  Generating report 'D:\src\DataExtraction\src\DataExtractionDotnet\BinaryAnalyzerCIL.Tests\coverage.json'

+--------------------------------+--------+--------+--------+
| Module                         | Line   | Branch | Method |
+--------------------------------+--------+--------+--------+
| BinaryAnalyzerCIL              | 80.99% | 84.61% | 79.48% |
+--------------------------------+--------+--------+--------+
| DataExtraction.GraphAnnotation | 70%    | 61.32% | 82.05% |
+--------------------------------+--------+--------+--------+
| DataExtractionCommon           | 44.27% | 52.38% | 34.61% |
+--------------------------------+--------+--------+--------+
| DataExtractionCommon.Tests     | 3.78%  | 100%   | 4.1%   |
+--------------------------------+--------+--------+--------+
| DataExtractorCIL          | 69.04% | 69.82% | 82.53% |
+--------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 59.83% | 66.5%  | 57.14% |
+---------+--------+--------+--------+
| Average | 53.61% | 73.62% | 56.55% |
+---------+--------+--------+--------+

[DhavalAlbiorix]

Hello @tonerdo
I'm also facing same issue with threshold, it's succeeded without any error. I'm using coverlet.msbuild 3.2.0.

[petli]

@linxingquan @DhavalAlbiorix This is most likely caused by the isse #1391, which is a problem with .NET SDK 7.0.0. That story discusses workarounds and links to the issue tracking the fix in the SDK. As an option, consider switching to the vstest integration which neither suffers this issue and is also recommended for CI environments: https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

[DhavalAlbiorix]

So, Does this support Threshold Validation? https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

…

On Thu, Dec 15, 2022 at 3:08 PM Peter Liljenberg ***@***.***> wrote: This is most likely caused by the isse #1391 <#1391>, and a problem with .NET SDK 7.0.0. That story discusses workarounds and links to the issue tracking the fix in the SDK. As an option, consider switching to the vstest integration which neither suffers this issue and is also recommended for CI environments: https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md — Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AU7F224E4NFEEQ3KXSMCNP3WNLRIDANCNFSM4E3AUYDA> . You are receiving this because you commented.Message ID: ***@***.***>

-- *Regards,* *Dhaval Thakral* *Sr. Software Engineer* Albiorix Technology Pvt. Ltd., Mobile: 7202013704

[petli]

@DhavalAlbiorix Sorry, I forgot about the threshold part, I was just thinking about the command line parameter passing. The vstest driver doesn't support thresholds, you would need to extend the CI script to process the reports from coverlet with e.g. https://reportgenerator.io/ and check that in the script.

[daveMueller]

@DhavalAlbiorix the issue with msbuild was fixed by the microsoft guys with sdk 7.0.101 that was released last week microsoft/vstest#4014. I just checked it with a repro from another related issue and it works fine. Please give it a try again.