mawosoft / benchmarkdotnet Goto Github PK

https://github.com/mawosoft/BenchmarkDotNet/blob/gh-pages/xrefmap.yml

See https://github.com/mawosoft/BenchmarkDotNet.Tests/tree/master/BDNParamDisposal
for parameter/argument creation and disposal in BDN host and in generated projects.

The generated code does not dispose the actually used parameters/arguments at all. Disposal occurs only while enumerating a ParamsSource or ArgumentsSource to skip to the current one and only for the ones skipped:

BDN host only disposes benchmarks it actually runs and only after they all have been run:

However, BenchmarkCase(s) - and therefore ParameterInstance(s) - have been already created early and have gone through filters and validators.

• If some are filtered out, they are not disposed.
• If any validator reports a critical error, none are disposed at all.

The problem is that ParameterInstance(s) are constructed per method. Multiple job definitions share the same cached instances. Thus a filtered BenchmarkCase cannot be disposed because it might share ParameterInstance(s) with benchmarks that are about to be run.

• ExecutionValidator(Base) has been written before introduction of Arguments and wasn't updated accordingly.
• ReturnValueValidator was written later(?), but is based on ExecutionValidatorBase => no ArgumentsSupport.
  It also uses InProcessNoEmitToolchain.FillMembers() which handles Params only.
• Both call WorkloadMethod.Invoke() with null for arguments.

Not sure if these validators are used anymore. They probably wouldn't have worked for our use case in XmlBenchmarks anyway due to lack of arguments equality (not sure about byte array equality, but ParamWrapper currently is in the way, see mawosoft/Mawosoft.Extensions.BenchmarkDotNet#22).

A bit problematic since SummaryStyle can be null which seems to mean SummaryStyle.Default.

Affects:

public static class BenchmarkConverter {
    public static BenchmarkRunInfo MethodsToBenchmarks(Type containingType, MethodInfo[] benchmarkMethods, IConfig config = null) {}
}
public static class BenchmarkRunner {
    public static Summary Run(Type type, MethodInfo[] methods, IConfig config = null) {}
} 

• BDN doesn't check if the given MethodInfos belong to the given Type and will convert the methods anyway.
• In case of a mismatch, the DisplayInfo will be wrong because BDN combines the class name from Type with the method name from MethodInfo despite the latter having a different ReflectedType (DeclaringType can be different if base class).
• For out-of-process jobs, building the auto-generated boilerplate code fails, because the Runnable_0 etc classes are derived from the wrong type.
• In-process jobs are executed w/o failing, but display the wrong info.

MemoryDiagnoser: MetricColumn "Allocated" always shows Bytes if SummaryStyle.PrintUnitsInContent is false, regardless of calculated best SizeUnit.

• This is independent from SummaryStyle.PrintUnitsInHeader being true or false.
• If PrintUnitsInHeader is true, it will show the correct calculated best SizeUnit in header, e.g. [KB], but the value is still in Bytes.

Introduced in 854633f

For Add() and Create() this is definitely a bug, however with Union() I'm not so sure. If localConfig was selected via LocalOnly, it might make sense to go back to Union for the resulting config???

Furthermore, Union() ALWAYS uses the UnionRule from localConfig.

• Why would a localConfig specify GlobalOnly?
• Setting GlobalOnly in the globalConfig has no effect. One would expect that to take precedence.

in SmartParameter.ToSourceCode():

Similar issue was fixed in SmartArgument.ToSourceCode():

-string cast = $"({Value.GetType().GetCorrectCSharpTypeName()})"; // it's an object so we need to cast it to the right type
+string cast = $"({parameterDefinitions[argumentIndex].ParameterType.GetCorrectCSharpTypeName()})"; // it's an object so we need to cast it to the right type

Would probably fix dotnet#2011

Almost all of these package refs are outdated:

  <ItemGroup>
    <PackageReference Include="CommandLineParser" Version="2.8.0" />
    <PackageReference Include="Iced" Version="1.14.0" />
    <PackageReference Include="Microsoft.Diagnostics.Runtime" Version="2.0.226801" />
    <PackageReference Include="Microsoft.DotNet.PlatformAbstractions" Version="3.1.6" />
    <PackageReference Include="Microsoft.Win32.Registry" Version="5.0.0" />
    <PackageReference Include="Perfolizer" Version="0.2.1" />
    <PackageReference Include="System.Management" Version="5.0.0" />
    <PackageReference Include="System.Reflection.Emit" Version="4.7.0" />
    <PackageReference Include="System.Reflection.Emit.Lightweight" Version="4.7.0" />
    <PackageReference Include="System.ValueTuple" Version="4.5.0" />
    <PackageReference Include="System.Threading.Tasks.Extensions" Version="4.5.4" />
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.11.0" />
    <PackageReference Include="Microsoft.Diagnostics.NETCore.Client" Version="0.2.236902" />
    <PackageReference Include="Microsoft.Diagnostics.Tracing.TraceEvent" Version="2.0.71" PrivateAssets="contentfiles;analyzers" />
  </ItemGroup>

Noticed this as BDN wasn't reporting VM infos with .NET core builds because System.Management.ManagementObjectSearcher throws an exception that's silently eaten.
The outdated refs might cause some other side effects as well.
Don't know if they are kept for some backward compat.