MSBuild Params Not Applied to Benchamrk

[rhys-wd]

I'm unable to apply MsBuildArguments to benchmarks, the toy solution below exhibits the bug

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;

var config = ManualConfig.Create(DefaultConfig.Instance)
		.WithOptions(ConfigOptions.DisableOptimizationsValidator);

BenchmarkSwitcher.FromAssemblies(
	new[] {
		typeof(Program).Assembly
	}
).Run(args, config);

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Jobs;

[Config(typeof(AConfig))]
public class A
{
	public class AConfig : ManualConfig
	{
		public AConfig()
		{
			AddJob(
				Job.Default
					.WithRuntime(CoreRuntime.Core60)
					.WithId("Baseline")
					.AsBaseline()
			);

			AddJob(
				Job.Default
					.WithArguments(new[] { new MsBuildArgument(@"-p:MY_PREPROCESSOR_DIRECTIVE=""true""") })
					.WithRuntime(CoreRuntime.Core60)
					.WithId("MY_PREPROCESSOR_DIRECTIVE")
			);
		}
	}

	[Benchmark]
	public void ScopeCommit()
	{
#if MY_PREPROCESSOR_DIRECTIVE
				#error MY_PREPROCESSOR_DIRECTIVE set
#endif

		var a = 1 + 1;
		a++;
	}
}

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net6.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="BenchmarkDotNet" Version="0.13.10" />
    </ItemGroup>


  <PropertyGroup Condition="'$(MY_PREPROCESSOR_DIRECTIVE)'=='true'">
    <DefineConstants>$(DefineConstants);MY_PREPROCESSOR_DIRECTIVE</DefineConstants>
  </PropertyGroup>
</Project>

If I run msBuild directly it compiles as expected dotnet build -p:MY_PREPROCESSOR_DIRECTIVE="true"

MSBuild version 17.7.4+3ebbd7c49 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
[...]
C:\code\Adaptive.Integration.Benchmark\A.cs(52,12): error CS1029: #error: 'MY_PREPROCESSOR_DIRECTIVE set' [C:\code\Benchmark\Benchmark.csproj]

Build FAILED.

C:\code\Benchmark\A.cs(52,12): error CS1029: #error: 'MY_PREPROCESSOR_DIRECTIVE set' [C:\code\Benchmark\Benchmark.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:06.05

[adamsitnik]

I am able to reproduce with the following code:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace BdnReproMsBuildArgs
{
    public class Program
    {
        static void Main(string[] args)
        {
            var config = ManualConfig.Create(DefaultConfig.Instance)
                .WithOptions(ConfigOptions.DisableOptimizationsValidator)
                .AddJob(Job.Dry
                        .WithArguments(new[] { new MsBuildArgument("/p:MY_PREPROCESSOR_DIRECTIVE=true") }));

            BenchmarkSwitcher
                .FromAssemblies(new[] { typeof(Program).Assembly })
                .RunAll(config, args);
        }

        [Benchmark]
        public void ScopeCommit()
        {
#if MY_PREPROCESSOR_DIRECTIVE
#error MY_PREPROCESSOR_DIRECTIVE set
#endif

            var a = 1 + 1;
            a++;
        }
    }
}

Let me take a quick look.

[adamsitnik]

OK, I've enabled the build output logging with:

.WithOptions(ConfigOptions.DisableOptimizationsValidator | ConfigOptions.LogBuildOutput)

In the log I can see following output:

// start dotnet  restore /p:MY_PREPROCESSOR_DIRECTIVE=true /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry
  Determining projects to restore...
  All projects are up-to-date for restore.
// command took 0.94 sec and exited with 0
// start dotnet  build -c Release /p:MY_PREPROCESSOR_DIRECTIVE=true --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry
MSBuild version 17.9.0-preview-23527-03+d3fa6693d for .NET
C:\Program Files\dotnet\sdk\9.0.100-alpha.1.23531.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry\BenchmarkDotNet.Autogenerated.csproj]
C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\Program.cs(26,8): error CS1029: #error: 'MY_PREPROCESSOR_DIRECTIVE set' [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\BdnReproMsBuildArgs.csproj]
Build FAILED.
C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\Program.cs(26,8): error CS1029: #error: 'MY_PREPROCESSOR_DIRECTIVE set' [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\BdnReproMsBuildArgs.csproj]
    0 Warning(s)
    1 Error(s)
Time Elapsed 00:00:01.46
// command took 1.73 sec and exited with 1
// start dotnet  build -c Release /p:MY_PREPROCESSOR_DIRECTIVE=true --no-restore --no-dependencies /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry
MSBuild version 17.9.0-preview-23527-03+d3fa6693d for .NET
C:\Program Files\dotnet\sdk\9.0.100-alpha.1.23531.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry\BenchmarkDotNet.Autogenerated.csproj]
  BenchmarkDotNet.Autogenerated -> C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\Dry\bin\Release\net6.0\Dry.dll
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:02.07

The interesting part is:

error: 'MY_PREPROCESSOR_DIRECTIVE set' [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\BdnReproMsBuildArgs.csproj]
Build FAILED.

So what BDN does is trying to perform full build (dependencies included), the build fails and it falls back to dotnet build --no-dependencies.

Should it just fail? Yes, but it would be hard on our side to distinguish such failures from actual build failures.

When I change the benchmark to simply do something else when the directive is set:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace BdnReproMsBuildArgs
{
    public class Program
    {
        static void Main(string[] args)
        {
            var config = ManualConfig.Create(DefaultConfig.Instance)
                .WithOptions(ConfigOptions.DisableOptimizationsValidator | ConfigOptions.LogBuildOutput)
                //.AddJob(Job.Dry)
                .AddJob(Job.Dry
                        .WithArguments(new[] { new MsBuildArgument("/p:MY_PREPROCESSOR_DIRECTIVE=true") }));

            BenchmarkSwitcher
                .FromAssemblies(new[] { typeof(Program).Assembly })
                .RunAll(config, args);
        }

        [Benchmark]
        public void ScopeCommit()
        {
#if MY_PREPROCESSOR_DIRECTIVE
            Thread.Sleep(TimeSpan.FromSeconds(10));
#else
            Thread.Sleep(TimeSpan.FromSeconds(1));
#endif
        }
    }
}

The build also fails, but with different error: the file is in use, so it can not be rebuilt:

// start dotnet  build -c Release /p:MY_PREPROCESSOR_DIRECTIVE=true --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\0f1ccdc3-be0f-4e43-8145-0afbee915107
MSBuild version 17.9.0-preview-23527-03+d3fa6693d for .NET
C:\Program Files\dotnet\sdk\9.0.100-alpha.1.23531.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\0f1ccdc3-be0f-4e43-8145-0afbee915107\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\9.0.100-alpha.1.23531.2\Microsoft.Common.CurrentVersion.targets(5198,5): warning MSB3026: Could not copy "C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\obj\Release\net6.0\apphost.exe" to "bin\Release\net6.0\BdnReproMsBuildArgs.exe". Beginning retry 1 in 1000ms. The process cannot access the file 'C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\bin\Release\net6.0\BdnReproMsBuildArgs.exe' because it is being used by another process. The file is locked by: "BdnReproMsBuildArgs (44656)" [C:\Users\adsitnik\source\repos\BdnReproMsBuildArgs\BdnReproMsBuildArgs.csproj]

What I can offer right now is a workaround: use WithCustomBuildConfiguration to set a build configuration:

.AddJob(Job.Dry.WithCustomBuildConfiguration("MY_PREPROCESSOR_DIRECTIVE"));

Once you do that you won't need the following logic:

<PropertyGroup Condition="'$(MY_PREPROCESSOR_DIRECTIVE)'=='true'">
    <DefineConstants>$(DefineConstants);MY_PREPROCESSOR_DIRECTIVE</DefineConstants>
</PropertyGroup>

Moreover, you can also remove ConfigOptions.DisableOptimizationsValidator as BDN is going to enable optimizations for all custom build configs except DEBUG

[rhys-wd]

Thanks for following up! I was trying to use the build failure as a sanity test and never assumed it'd be swallowed.

Regardless, thank you so much!

[timcassell]

This will be fixed by #2393.