BenchmarkDotNet Out-of-box experience and "getting started" documentation about benchmarkdotnet HOT 21 CLOSED

“Getting started” is a great idea, we definitely should have such section.
We can replace "Why is microbenchmarking hard?" with the link to http://andreyakinshin.gitbooks.io/performancebookdotnet/content/science/microbenchmarking.html

from benchmarkdotnet.

@mattwarren, I have updated the README. What do you think?

from benchmarkdotnet.

My thoughts on getting started (and apologies if I didn't follow the readme correctly):

from benchmarkdotnet.

@smudge202 how exactly are you executing the tests?

From reading this comment I assume that you are using DNX, is that right?

@AndreyAkinshin has only recently added support for CoreCLR, so maybe he can comment a bit more about this.

from benchmarkdotnet.

The CoreCLR support will be added after Microsoft Connect(); 2015, I'm waiting for some updates. =)

from benchmarkdotnet.

Do you know if the issue that @smudge202 is getting is related to DNX/CoreCLR support?

Or do you think it's a general problem because his Benchmark look like this:

[Benchmark]
public void DispatchingToNoReceivers()
    => Dispatcher.Dispatch(Event);

and we don't currently support that type of Benchmark?

from benchmarkdotnet.

As per the comment you referenced, @mattwarren , I'm fairly sure it'll be a DNX incompatibility. Unfortunately, the NuGet package doesn't list supported frameworks.

from benchmarkdotnet.

Ignore my comment above, I just ran a test and we do support [Benchmark] methods created with the new C# 6 syntax, i.e. this works fine:

[Benchmark]
public void SleepWithCSharp6Syntax() => Thread.Sleep(100);

from benchmarkdotnet.

@mattwarren, I believe you're confusing DNX/CoreCLR with C#6. Two very different things.

C#6 is independent, simply something the compiler does or does not support.

from benchmarkdotnet.

@AndreyAkinshin I just saw this post The road to DNX – part 1 and thought you might find it useful when you do the DNX/CoreCLR work.

Update: The road to DNX – part 2 and The road to DNX – part 3

from benchmarkdotnet.

I've taken a look at this issue and published my WIP branch here.

The actual issue is of course an incompatibility with vNext projects (or whatever name we're going by today), but I went a step further and converted the existing projects too (hopefully that doesn't present too much confusion, but it does prove that even the existing tests fail when hosted by DNX). I actually wanted to see if we could get a netstandard1.*/dotnet5.*+1 TFM in there as well, but I think the third party and framework dependencies will prevent that.

Anyway, the specific issue is that whilst running under DNX (such as through the xunit.runner.dnx as shown in the linked fork because I've converted the existing tests to run under DNX), you don't get as much information or even the same information relating to assemblies. Assembly paths will typically return the location of dnx.exe, whilst Assembly.Location will return null, as is the case on this line causing a path is not of a legal form exception on line 155..

To work around this, I previously used this stackoverflow answer. Here lies the issue, the current API handles all assembly/file/type handling internally and I see no obvious extensibility points whereby DNX could be plugged in to the core library allowing the location to be retrieved correctly based upon the underlying runtime.

I could take a stab at making BenchmarkDotNet more extensible in order to allow such a plugin mechanism, though it may well be easier to cross compile the core code with separate BenchmarkClassicGenerator classes based upon TFM and wrap them uptogether in the same nuget package? I don't want to sink too much time into this without someone up the chain taking a peak and pointing me in the direction you'd like to go (even if that direction is to buzz off :) )

from benchmarkdotnet.

@smudge202,

I could take a stab at making BenchmarkDotNet more extensible in order to allow such a plugin mechanism

I'll be grateful for any help! CoreCLR support is a very important feature, we need it in order to measure efficiency of new version of RyuJIT.

About the architecture: now we have the following interfaces: IBenchmarkGenerator, IBenchmarkBuilder, IBenchmarkExecutor, and the "classic" implementations of it. I want a new "DNX" implementations (or which name should we choose?). Next, BenchmarkToolchainFacade should be able to create a facade for new implementation in the CreateToolchain method. The BenchmarkToolchain enum should have a new value (like Dnx) which we will use in BenchmarkTasks. Ideally, the DNX support shouldn't affect to other parts of the library (except Generator/Builder/Executor).

Feel free to ask any details (we can discuss the subject in a separate issue or in the chat).

from benchmarkdotnet.

Thanks for the information, @AndreyAkinshin . To be clear, I believe CoreCLR support and DNX support are going to be two different issues.

To move to CoreCLR (and take advantage of what I hear is coming with rc2 in February), I think we need to find a way to drop the dependencies on System.Management, Microsoft.Diagnostic.Runtime, Dia2Lib, and Interop.Dia2Lib. To be honest, apart from seeing that we reference them, I haven't see how deeply nested the dependency goes. I recommend opening a second issue to tackle CoreCLR.

With regards to DNX, I actually took a look at implementing a cross compiled version last night. I defined DNX (which I think David Fowler hates...) into a dnx451 target, pulled in a Microsoft.Dnx.Runtime dependency for the dnx target, and started going through all the interfaces available through the common service locator. We can get access to the LibraryManager, RuntimeEnvironment and ApplicationEnvironment, however, none of those expose the assembly location. Thinking about it, that is probably by design. I won't speculate with regards to my very poor understanding on how DNX compiles (on the fly?) and how assembly location probably doesn't make sense.

I think there are two options. Ask guys from the DNX team (hop into their gitter or @ them on here) what they would recommend (should probably leave that as a last resort). Alternatively, examine why it is we need (or think we need) the assembly location. I only briefly glanced at the consumer code, but it almost looked like it was trying to generate a csproj style reference?

I totally hijacked a legitimate issue - should we hop onto a new issue for DNX support?

from benchmarkdotnet.

I think we need to find a way to drop the dependencies on System.Management, Microsoft.Diagnostic.Runtime, Dia2Lib, and Interop.Dia2Lib

@mattwarren, can you move these dependencies from the core to another assembly in the nearest future?

from benchmarkdotnet.

I think we need to find a way to drop the dependencies on System.Management, Microsoft.Diagnostic.Runtime, Dia2Lib, and Interop.Dia2Lib

Yeah no problem, in the next few days I'll check-in some changes so that these are moved out of BenchmarkDotNet core and put into "BenchmarkDotNet.Diagnostic" as we discussed.

from benchmarkdotnet.

In The road to DNX - part 2, Marc says this:

I do not recommend polluting all your code with constant #if. Rather, my strategy is to create a utility class that bridges the gap (usually, but not always, via extension methods), and have just that class deal with different implementations.

It would be nice if we could follow this advice as it seems a lot cleaner to have all the #if code in as few places as possible. (not saying you were going to do this, just wanted flag it up)

from benchmarkdotnet.

Yup. I usually ifdef to get things running and then look to clean things up. Happy to go with extensions if/when we move forward on this.

from benchmarkdotnet.

@mattwarren, are you happy with current version of README?

from benchmarkdotnet.

It's been a while since I read through it all the way, I'll take a look in the next day or so and if I spot anything I'll make some changes.

from benchmarkdotnet.

Great, thanks.

from benchmarkdotnet.

Now we have nice documentation with the "Getting started" section. We also have another issue about online documentation: #219

from benchmarkdotnet.