NativeMemoryProfiler exception

[WojciechNagorski]

@dotMorten has reproduced an exception in NativeMemoryProfiler.
There is https://twitter.com/dotMorten/status/1166842655348670466

At first, he got an error:

Then:

Setting ArtifactsPath="./" in my manual config seems to work (null check missing there?). Now it runs for a lot longer and creates .etl files, then goes boom here:

[adamsitnik]

@wojtpl2 are you able to repro it?

[WojciechNagorski]

@adamsitnik No, I'm not.
@dotMorten Could you share some reproduction?

I have some ideas:

• Copy of native DLL
  Related with Copy to $(OutDir) does not copy to benchmarkdotnet output folder #946 . I know good solution for it. Instead of using AfterBuild, use this construction: https://github.com/wojtpl2/NativeMemorySample/blob/master/NativeDllWraper/NativeDllWraper.csproj#L6-L9. I checked that this solution worked even for the directory. I'm writing a blog post about it.
• Working without admin?
• Use this NativeMemoryProfiler with another profiler which uses EtwProfiler.

[WojciechNagorski]

the first exception occurred in this line:

BenchmarkDotNet/src/BenchmarkDotNet.Diagnostics.Windows/Sessions.cs

Line 147 in e92abf8

return Path.Combine(details.Config.ArtifactsPath, $"{fileName}{FileExtension}");

It is possible only if details.Config.ArtifactsPath is null. I need time to understand when this can be null.
@dotMorten If you are not able to share reproduction, please share only your configuration. I mean csproj and benchmark configuration.

[dotMorten]

I should be able to share something once I get into the office in a bit. The second crash only occurs for some tests, and the etl file was huuuuge at that point (over 8gb). Other tests worked fine and detected native memory.
Only see two places where division by error can occur, but not sure why those values would be zero at that point.

[WojciechNagorski]

I've just pushed nicer error handling. Division by zero is possible only when *.etl file doesn't contain events from BenchmarkDotNet Engine. To be honest, I don't know yet why it works that way, but I will find this bug 😄

[dotMorten]

Below is a repro for the artifacts path. I'll get back to you with the other issue, once I zero in on it (the one that fails is overly complicated so trying to dumb it down):

You'll need this package reference:

    <PackageReference Include="Esri.ArcGISRuntime.WPF">
      <Version>100.6.0</Version>
    </PackageReference>

Repro app code:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains.CsProj;
using Esri.ArcGISRuntime.Geometry;
using System;
using System.Linq;

namespace BDN_Repro
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new MyConfig();
            var summary = BenchmarkRunner.Run(typeof(GeometryBuilderBenchmarks), config);
        }
    }
    public class GeometryBuilderBenchmarks
    {
        MapPointBuilder pointbuilder;
        [GlobalSetup]
        public void Setup()
        {
            pointbuilder = new MapPointBuilder(123, 456, SpatialReferences.Wgs84);
        }

        [Benchmark]
        public double MapPointBuilder_InteropOverhead_GetSetXY()
        {
            return ((pointbuilder.X += 1) + (pointbuilder.Y += 1));
        }
    }

    public class MyConfig : ManualConfig
    {
        private Job baseJob;
        public MyConfig()
        {
            baseJob = Job.MediumRun.With(CsProjClassicNetToolchain.Net48);


            Add(DefaultConfig.Instance.GetLoggers().ToArray());
            Add(DefaultConfig.Instance.GetExporters().ToArray());
            Add(DefaultConfig.Instance.GetColumnProviders().ToArray());
            Add(BenchmarkDotNet.Diagnosers.MemoryDiagnoser.Default);
            Add(new BenchmarkDotNet.Diagnostics.Windows.NativeMemoryProfiler());
            // ArtifactsPath = "./"; //This fixes the crash
        }
    }
}

[WojciechNagorski]

@dotMorten Thanks for this reproduction. I've just pushed fix.

The problem was not in NativeMemoryProfiler.

When you run benchmark with your own configuration in this way:

        static void Main(string[] args)
        {
            var config = new MyConfig();
            var summary = BenchmarkRunner.Run(typeof(GeometryBuilderBenchmarks), config);
        }

Then resulted ArtifactsPath is null.

I found information in the code that in this case the default path should be used.

BenchmarkDotNet/src/BenchmarkDotNet/Validators/ConfigValidator.cs

Lines 41 to 43 in 3bd18fc

	private static ValidationError ValidateArtifactsPath(string artifactsPath)
	{
	if (artifactsPath == null) // null is OK, default path will be used

I check that the null value in ArtifactsPath causes an exception for all EtwProfiler, for instance, the configuration below raises the same exception:

        public class MyConfig : ManualConfig
        {
            private Job baseJob;
            public MyConfig()
            {
                baseJob = Job.MediumRun.With(CsProjClassicNetToolchain.Net461);


                Add(DefaultConfig.Instance.GetLoggers().ToArray());
                Add(DefaultConfig.Instance.GetExporters().ToArray());
                Add(DefaultConfig.Instance.GetColumnProviders().ToArray());
                Add(BenchmarkDotNet.Diagnosers.MemoryDiagnoser.Default);
                Add(new BenchmarkDotNet.Diagnostics.Windows.EtwProfiler()); // <-- Use EtwProfiler instead of NativeMemoryProfiler
            }
        }

[dotMorten]

@wojtpl2 I think what I'm seeing with the second crash is because the test threw an exception in a background thread.

[WojciechNagorski]

@adamsitnik @dotMorten I think that PR #1237 is ready to review and this issue is ready to close.

I checked that exception in a background thread stops benchmark and changes from #1237 shows the message:

[adamsitnik]

Fixed in #1237 a while ago, but we forgot to close the issue. So I am closing it now.