Comments (8)
Hi,
@adamsitnik, @AndreyAkinshin, can I work on this issue as my first one? I performed two simple benchmark tests on my laptop. Below you can see, it is a significant difference between power plans.
private readonly Dictionary<string, string> Guids = new Dictionary<string, string>()
{
{ "HighPerformance", "8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c" },
{ "Balanced", "381b4222-f694-41f0-9685-ff5bb260df2e" },
{ "PowerSaver", "e22fd527-0c09-43ad-83d6-ba300affc27d" }
};
[Params("HighPerformance", "Balanced", "PowerSaver")]
public string PerfPlan { get; set; }
[IterationSetup]
public void InitGuid()
{
var planGuid = Guids[PerfPlan];
var guid = new Guid(planGuid);
PowerManagment.Set(guid);
}
[Benchmark]
public int SplitJoin()
=> string.Join(",", new string[1000]).Split(',').Length;
[Benchmark]
public int IterationTest()
{
int j = 0;
for (int i=0; i < short.MaxValue; ++i)
{
j = i;
}
return j;
}
Method | PerfPlan | Mean | Error | StdDev | Median |
---|---|---|---|---|---|
SplitJoin | Balanced | 32.09 us | 0.8718 us | 2.2348 us | 33.32 us |
IterationTest | Balanced | 24.28 us | 0.7690 us | 2.1435 us | 25.38 us |
SplitJoin | HighPerformance | 28.58 us | 0.5693 us | 1.0553 us | 27.89 us |
IterationTest | HighPerformance | 20.79 us | 0.1932 us | 0.1807 us | 20.68 us |
SplitJoin | PowerSaver | 70.12 us | 0.7759 us | 0.7258 us | 69.99 us |
IterationTest | PowerSaver | 55.01 us | 0.0000 us | 0.0000 us | 55.01 us |
What I suggest that we can use helper class similar to that proposed by @omariom with two methods. The first one will check if the system is windows, save current power plan and eventually set performance to the HighPerf. Then we perform all benchmarks, and after that, we use the second method. The second method will restore the previous user power plan. Both of the methods will execute in the BenchmarkRunner class. What is your opinion about that?
Points to consider:
If someone shutdown system or kill benchmark process, unfortunately, a user will stay with a new performance plan as the side effect of benchmarking. Of course, we can log some information to the user that we changed his power plan.
from benchmarkdotnet.
It is a very good idea, I like it. Another good links about the subject:
https://msdn.microsoft.com/en-us/library/system.windows.forms.systeminformation.aspx
https://msdn.microsoft.com/en-us/library/hxkc1kwd.aspx
http://thoughtso.org/wp-content/uploads/2007/11/cpower.txt
http://stackoverflow.com/questions/16023978/how-to-use-power-management-functions-powerenuimerate-to-get-power-settings
http://www.fsmpi.uni-bayreuth.de/~dun3/archives/programmatically-change-power-options-using-cshar/519.html
from benchmarkdotnet.
Agree, that's a great idea, I've used Vance Morrison's MeasureIt before, but I never realised that it forced the power mode like that.
from benchmarkdotnet.
Some more related info (from a comment on Scott Hanselman's Blog Post "Benchmarking .NET code"):
We benchmark our code all the time. For serialization we have done this: SERBENCH here are some charts: Typical Person Serialization.
What is interesting is that the most benchmarks provided by software makers are very skewed, i.e. serializers depend greatly on payload type, as our test indicated. Some things taken for granted like "Protobuff" is the fastest are not true in some test cases you see unexpected results. That's is why it is important to execute the same suite of test cases against different vendors. Sometimes benchmarking something is more work than writing the component, for example SpinLock class has many surprises.Another thing that 90% of people I talk to disregard: SpeedStep. Try to disable it via control panel and dont be surprised that you get 25% perf boost EVEN IF you ran your tests for minutes. SpeedStep does magic tricks, it lowers you CPU clock even if the CPU is swamped, but not on all cores. Running tests from within VS(with debugger attached) can slow things down by good 45-70%.
So I would say:
- Test different work patterns
- Build with Optimize
- Run standalone (CTRL+f5)
- Disable "Power Saver"/SpeedStep
- Run for a reasonable time (not 2 seconds)
- For multi-threaded perf dont forget to set GS SERVER MODE, otherwise your "new" will be 10-40% slower than it could
- Profilers sometimes lie! They don't guarantee nothing. Great tools but sometimes they don't show whats happening and may be source of Heisenbugs
from benchmarkdotnet.
there was no progress for 8 months, so I made it up for grabs again
from benchmarkdotnet.
can I work on this issue as my first one?
Sure!
from benchmarkdotnet.
What I suggest that we can use helper class similar to that proposed by @omariom with two methods. The first one will check if the system is windows, save current power plan and eventually set performance to the HighPerf. Then we perform all benchmarks, and after that, we use the second method. The second method will restore the previous user power plan. Both of the methods will execute in the BenchmarkRunner class. What is your opinion about that?
Looks good to me.
Also, we need an option in Config which allows disabling this behavior.
from benchmarkdotnet.
Fixed in #952
from benchmarkdotnet.
Related Issues (20)
- Support for F# anonymous records HOT 3
- MonoAOTLLVM run failed to build properly but still ran HOT 2
- Add integration tests for `MonoAOTLLVMToolchain`
- Unable to get `-p ETW` working these days HOT 4
- Unable to use `wasm` or `monoaotllvm` as runtime HOT 2
- InvalidMeasurementUnitException when using `--statisticalTest` argument when comparing private builds HOT 2
- CS0119 thrown when benchmark named _.System()
- How to change the time display mode from microseconds to seconds or milliseconds? HOT 1
- HideColumns + custom corerun = incorrect table markdown HOT 2
- unity game engine support? HOT 1
- Cannot build NativeAOT 8.0 on Arch Linux HOT 6
- Add support for dotnet-script HOT 1
- What should I do if I need to see the output of each test item? HOT 2
- What should I do if I need to see the output of each test item? HOT 1
- Plots not generated for memory benchmarks (MemoryDiagnoser)
- Question about visibility of DotTraceDiagnoser's second constructor HOT 2
- BenchmarkDotNet does not copy needed assemblies to the run folder (again?) HOT 2
- Can't override `DefaultConfig.Instance` settings when using custom configs with `IConfigSource` attribute.
- Better support for asymmetric CPU cores HOT 3
- Total Number of Benchmarks
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarkdotnet.