Replace ConditionalFact usages with custom facts #2402

mareklinka · 2019-02-04T17:39:12Z

As proposed in #1220, this PR replaces all usages of ConditionalFact in tests with custom-tailored, environment-specific facts.

These specific facts all inherit from a single base EnvironmentSpecificFactAttribute. Several of the facts check the same condition (e.g. TensorFlow, Onnx, and X64 - they all simply check for X64 process) but I decided not to merge them - they might check the same thing now but I expect they might branch out later on. I consider the small duplication of code not worth the mental overhead of inheritance or composition here.

Together with this I also removed some now-obsolete checking of platform in Onnx-related tests (they were all Windows-only anyway) and removed the old OnnxFact.

Fixes #1220.

mareklinka · 2019-02-04T19:57:54Z

Ok, so it seems I stumbled upon an interesting problem here. It seems tests marked with ConditionalAttribute were not running at all under full framework. Now that I migrated them to custom attributes, they are being run and fail.

Specifically, the failing Onnx tests are looking for the model files in the wrong directory. Debugging into it, this is the path is returned in the ResNet18 extension method:

C:\Users\Marek\AppData\Local\Temp\4e765c63-f6ab-4125-b0da-31d31f556eb7\4e765c63-f6ab-4125-b0da-31d31f556eb7\assembly\dl3\dd76d9c2\39935ac9_c0bcd401\DnnImageModels

This feels like some kind of shadow copy is going on here.

I'm still looking into the tensorflow and MacOS fail.

mareklinka · 2019-02-04T20:33:20Z

The TensorFlow failure seems to be of the same root cause - it was not running under full framework, now it runs. However, the directory Maml.MainWithProgress looks at is wrong:

C:\Users\Marek\AppData\Local\Temp\bd281e66-574e-4cfc-b712-1768a003416c\bd281e66-574e-4cfc-b712-1768a003416c\assembly\dl3\c6759ee3\bfbd336c_c7bcd401

I have half a mind to make EnvironmentSpecificFactAttribute always return false #if NETFRAMEWORK to emulate the old behavior (not running these tests on full framework). Disabling shadow copying might be an option here but I'm unfamiliar with the build setup so I don't know where to look.

Could someone point me in the right direction, @tannergooding, @eerhardt?

eerhardt · 2019-02-04T20:34:53Z

@Anipik has been looking in this same space. Adding him to comment.

TomFinley · 2019-02-04T20:49:01Z

Yes. Also given the subject area (TensorFlow components) I would also invite @jignparm to possibly share his thoughts.

test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

Anipik · 2019-02-04T20:54:48Z

I have half a mind to make EnvironmentSpecificFactAttribute always return false #if NETFRAMEWORK to emulate the old behavior (not running these tests on full framework)

I would prefer to disable these (failing onnx test) tests just on fullFramework, using notFullFrameworkFact or NotFullFrameworkAndx64. As most of the tests are passing using conditional Fact are passing on fullFramework.

Anipik · 2019-02-05T00:45:23Z

I am able to fix all the resnet failures in #2411

mareklinka · 2019-02-05T08:25:05Z

Awesome news on the test fixes, @Anipik. I'll wait until your PRs are merged and then rebase my branch onto the latest master.

I did some small refactorings (e.g. bring the attribute class names in line and add AttributeUsage) and also replaced the usages of ConditionalTheory. The LessThanNetCore30OrNotNetCoreFactAttribute now requires a skip message so it's obvious at usage that this is something that's non-standard and should be fixed down the line.

codecov · 2019-02-05T08:54:17Z

Codecov Report

Merging #2402 into master will increase coverage by <.01%.
The diff coverage is 94.33%.

@@            Coverage Diff             @@
##           master    #2402      +/-   ##
==========================================
+ Coverage   71.24%   71.24%   +<.01%     
==========================================
  Files         788      797       +9     
  Lines      141131   141162      +31     
  Branches    16115    16111       -4     
==========================================
+ Hits       100545   100572      +27     
- Misses      36121    36127       +6     
+ Partials     4465     4463       -2

Flag	Coverage Δ
#Debug	`71.24% <94.33%> (ø)`	⬆️
#production	`67.57% <ø> (ø)`	⬆️
#test	`85.35% <94.33%> (-0.01%)`	⬇️

mareklinka · 2019-02-05T13:33:04Z

Now the Core 3.0 Debug intrinsics build failed. Looking into it, it tried to run the benchmarks in DEBUG build, which should not have happened - the BenchmarkTheoryAttribute uses the same #if directives as the original - it should skip #if DEBUG. It looks like the DEBUG constant was not emitted during compilation of the test project?

Anipik

can you also remove this https://github.com/dotnet/machinelearning/blob/master/test/Directory.Build.props#L32 ?

test/Microsoft.ML.TestFramework/Attributes/EnvironmentSpecificTheoryAttribute.cs

test/Microsoft.ML.TestFramework/Attributes/EnvironmentSpecificFactAttribute.cs

eerhardt · 2019-02-07T23:25:30Z

test/Microsoft.ML.Core.Tests/UnitTests/TestEntryPoints.cs

@@ -1313,7 +1314,7 @@ public void EntryPointMulticlassPipelineEnsemble()
            }
        }

-        [ConditionalFact(typeof(BaseTestBaseline), nameof(BaseTestBaseline.LessThanNetCore30OrNotNetCore))]
+        [LessThanNetCore30OrNotNetCoreFact("netcore3.0 output differs from Baseline")]


netcore3.0 => netcoreapp3.0. We should use the real TFM here, so we don't confuse people.

Repeated for all uses.

Anipik · 2019-02-08T00:37:59Z

@eerhardt I tried to debug the benchmarkTheory failure. For some reason Debug Constant is not working in this PR. I checked the Binlog and it is getting defined.

Property reassignment: $(DefineConstants)=";DEBUG;" (previous value: ";DEBUG") at C:\Program Files\dotnet\sdk\3.0.100-preview-010184\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.CSharp.props (22,5)

I tried adding it to the project as well but it didn't work. any suggestions why ?

mareklinka · 2019-02-08T10:23:47Z

can you also remove this https://github.com/dotnet/machinelearning/blob/master/test/Directory.Build.props#L32 ?

The issue with this is that that particular nuget also contains the definition of TestCategory. I can get rid of that but we would need to replace the TestCategory with XUnit's own Trait:

[TestCategory(Cat)]
[Fact(Skip = "Need CoreTLC specific baseline update")]
public void CommandCrossValidation()
{ ... }

would become

[Trait("Category", Cat)]
[Fact(Skip = "Need CoreTLC specific baseline update")]
public void CommandCrossValidation()
{ ... }

Alternatively, we could use the XUnit.Categories nuget and end up with something cleaner:

[Category(Cat)]
[Fact(Skip = "Need CoreTLC specific baseline update")]
public void CommandCrossValidation()
{ ... }

This nuget also contains some other useful attributes, like Feature, Bug, Work Item etc. What do you think, @Anipik, @eerhardt?

eerhardt · 2019-02-08T16:41:44Z

This nuget also contains some other useful attributes, like Feature, Bug, Work Item etc. What do you think, @Anipik, @eerhardt?

Let's keep using the Microsoft.DotNet.XUnitExtensions package. It still provides value.

eerhardt · 2019-02-08T17:02:29Z

I tried adding it to the project as well but it didn't work. any suggestions why ?

The #if DEBUG check is now happening in the Microsoft.ML.TestFramework project. Which means you have to look to see what defines are being used when building that project.

Looking at the .csproj, when we do things like this:

machinelearning/test/Microsoft.ML.TestFramework/Microsoft.ML.TestFramework.csproj

Line 3 in 834e471

<DefineConstants>CORECLR</DefineConstants>

That will completely override anything that is set earlier in a .props file. The correct line here is:

<DefineConstants>$(DefineConstants);CORECLR</DefineConstants>

But even better would be if we deleted this line from the .csproj all together and removed any dead code that is behind #if !CORECLR in this project.

mareklinka · 2019-02-08T17:08:46Z

But even better would be if we deleted this line from the .csproj all together and removed any dead code that is behind #if !CORECLR in this project.

Sounds good. I'll take a look at this over the weekend, see what falls out. To clarify, we are talking specifically the Microsoft.ML.TestFramework project here, right?

eerhardt · 2019-02-08T17:39:20Z

To clarify, we are talking specifically the Microsoft.ML.TestFramework project here, right?

Yes. The DEBUG define is not working in that project apparently. Just that project needs to be fixed now. I'm sure others have the same problems, but let's only do this one for now.

Add AttributeUsage attributes

…kip message

mareklinka · 2019-02-10T14:52:08Z

test/Microsoft.ML.TestFramework/DataPipe/TestDataPipeBase.cs

@@ -1096,23 +1096,6 @@ protected Func<bool> GetColumnComparer(Row r1, Row r2, int col, ColumnType type,
                    return GetComparerVec<RowId>(r1, r2, col, size, (x, y) => x.Equals(y));
            }

-#if !CORECLR // REVIEW: Port Picture type to CoreTLC.
-            if (type is PictureType)


I think this could have been ported from PictureType to ImageType, if I understand the problem correctly? Would require a reference to ImageAnalytics, though.

Currently we are only building the project with CORECLR defined. So this is dead code that can be deleted.

Let's just delete it for now. If @Ivanidzo4ka thinks this is important, we can open a new issue to add image support here. Thoughts @Ivanidzo4ka?

Feel free to delete. Some legacy code which we don't care about anymore

eerhardt · 2019-02-10T22:06:03Z

test/Microsoft.ML.TestFramework/Attributes/NotFullFrameworkFactAttribute.cs

+    /// <summary>
+    /// A fact for tests requiring full .NET Framework.
+    /// </summary>
+    public sealed class NotFullFrameworkFactAttribute : EnvironmentSpecificFactAttribute


Is this used? I don't see any usages doing a simple search.

eerhardt

Just one comment on a class that doesn't appear to be used. Other than that, this looks good. Thanks for tackling this - I think it really cleans the tests up nicely.

mareklinka · 2019-02-10T22:15:03Z

Just one comment on a class that doesn't appear to be used. Other than that, this looks good. Thanks for tackling this - I think it really cleans the tests up nicely.

Got rid of the unused attribute. Glad to be of help 😄

Anipik

LGTM

mareklinka force-pushed the feature/1220_ConditionalFactCleanup branch from e5e4ecb to f013125 Compare February 4, 2019 17:48

eerhardt requested a review from Anipik February 4, 2019 20:34

Anipik reviewed Feb 4, 2019

View reviewed changes

test/Microsoft.ML.Predictor.Tests/TestPredictors.cs Outdated Show resolved Hide resolved

Anipik mentioned this pull request Feb 4, 2019

Fix loading type problem for netfx #2409

Merged

This was referenced Feb 5, 2019

Add net461 to targetFrameworksList dotnet/arcade#1961

Closed

ResNet Tests corrected for netfx #2411

Merged

mareklinka force-pushed the feature/1220_ConditionalFactCleanup branch from bcc7bae to 8d75ef1 Compare February 6, 2019 14:59

Anipik reviewed Feb 7, 2019

View reviewed changes

test/Microsoft.ML.TestFramework/Attributes/EnvironmentSpecificTheoryAttribute.cs Show resolved Hide resolved

test/Microsoft.ML.TestFramework/Attributes/EnvironmentSpecificFactAttribute.cs Show resolved Hide resolved

eerhardt reviewed Feb 7, 2019

View reviewed changes

mareklinka added 5 commits February 10, 2019 15:48

Replace ConditionalFact usages with custom facts

0b200e7

Refactor ConditionalTheory usages

ae4399a

Add AttributeUsage attributes

Make LessThanNetCore30OrNotNetCoreFactAttribute require an explicit s…

45c6116

…kip message

Use correct target framework moniker for netcoreapp3.0

beb537a

Remove CORECLR constant

bc7b87b

mareklinka force-pushed the feature/1220_ConditionalFactCleanup branch from b1c8011 to bc7b87b Compare February 10, 2019 14:49

mareklinka commented Feb 10, 2019

View reviewed changes

eerhardt reviewed Feb 10, 2019

View reviewed changes

eerhardt approved these changes Feb 10, 2019

View reviewed changes

Remove an unused fact attribute

b6a5501

Anipik approved these changes Feb 11, 2019

View reviewed changes

Anipik merged commit 8444a5a into dotnet:master Feb 11, 2019

ghost locked as resolved and limited conversation to collaborators Mar 24, 2022

Replace ConditionalFact usages with custom facts #2402

Replace ConditionalFact usages with custom facts #2402

Uh oh!

Conversation

mareklinka commented Feb 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mareklinka commented Feb 4, 2019

Uh oh!

mareklinka commented Feb 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eerhardt commented Feb 4, 2019

Uh oh!

TomFinley commented Feb 4, 2019

Uh oh!

Uh oh!

Anipik commented Feb 4, 2019

Uh oh!

Anipik commented Feb 5, 2019

Uh oh!

mareklinka commented Feb 5, 2019

Uh oh!

codecov bot commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mareklinka commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anipik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eerhardt Feb 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anipik commented Feb 8, 2019

Uh oh!

mareklinka commented Feb 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eerhardt commented Feb 8, 2019

Uh oh!

eerhardt commented Feb 8, 2019

Uh oh!

mareklinka commented Feb 8, 2019

Uh oh!

eerhardt commented Feb 8, 2019

Uh oh!

mareklinka Feb 10, 2019

Choose a reason for hiding this comment

Uh oh!

eerhardt Feb 10, 2019

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Feb 10, 2019

Choose a reason for hiding this comment

Uh oh!

eerhardt Feb 10, 2019

Choose a reason for hiding this comment

Uh oh!

eerhardt left a comment

Choose a reason for hiding this comment

Uh oh!

mareklinka commented Feb 10, 2019

Uh oh!

Anipik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mareklinka commented Feb 4, 2019 •

edited

Loading

mareklinka commented Feb 4, 2019 •

edited

Loading

codecov bot commented Feb 5, 2019 •

edited

Loading

mareklinka commented Feb 5, 2019 •

edited

Loading

eerhardt Feb 7, 2019 •

edited

Loading

mareklinka commented Feb 8, 2019 •

edited

Loading