Any consideration of having benchmarks? about bepuphysics2 HOT 14 OPEN

Is there an easy way to tell which of the 3 categories they belong to?

Now there is!

from bepuphysics2.

Yup- there should only be 39 benchmarks in the trimmed set, which includes:

-f *CollisionBatcherTaskBenchmarks.* *GroupedCollisionTesterBenchmarks.* *GatherScatterBenchmarks.* *OneBodyConstraintBenchmarks.* *TwoBodyConstraintBenchmarks.* *ThreeBodyConstraintBenchmarks.* *FourBodyConstraintBenchmarks.* *SweepBenchmarks.* *ShapeRayBenchmarks.* *ShapePileBenchmark.* *RagdollTubeBenchmark.*

The deep dive set should include 123 benchmarks and takes a long time:

-f *CollisionBatcherTaskBenchmarks.* *ConvexCollisionTesterBenchmarks.* *GatherScatterBenchmarks.* *OneBodyConstraintBenchmarks.* *OneBodyConstraintBenchmarksDeep.* *TwoBodyConstraintBenchmarks.* *TwoBodyConstraintBenchmarksDeep.* *ThreeBodyConstraintBenchmarks.* *FourBodyConstraintBenchmarks.* *SweepBenchmarks.* *SweepBenchmarksDeep.* *ShapeRayBenchmarksDeep.* *ShapePileBenchmark.* *RagdollTubeBenchmark.*

Looks like that run included every benchmark unconditionally (128).

from bepuphysics2.

I see it to be 12 minutes.

Global total time: 00:12:01 (721.7 sec), executed benchmarks: 39

from bepuphysics2.

not sure what is the problem. I am able to run this fine.

from bepuphysics2.

Should be pretty easy; I'd be willing to toss some together.

A few questions:

What kind of scope would be appropriate for each test? For example, I could do:

1. Narrow single-codepath tests (e.g. one benchmark for BoxPairTester, another for BoxCylinderTester...)
2. Bundled tests covering large chunks of low level features (e.g. a bunch of collision tests, or a bunch of constraints in one benchmark)
3. Full simulation benchmarks

(I could do all of the above pretty easily if desired.)

I assume purely single threaded benchmarks would be more relevant for runtime purposes. Would there be any value in internally multithreaded benchmarks?

Is there anything else I should keep in mind when setting up the benchmarks to make including them easier?

from bepuphysics2.

Thank you @RossNordby for the quick and prompt response. Yes, all of the above categories looks good and it will help us keep track of individual components as well as broader scenarios.

I assume purely single threaded benchmarks would be more relevant for runtime purposes

That's correct.

Would there be any value in internally multithreaded benchmarks?

I don't think so for Microbenchmarks perspective. We can certainly do it in future as a standalone benchmarks.

Is there anything else I should keep in mind when setting up the benchmarks to make including them easier?

I think if you can make it runnable using BDN, that should be good enough. If you are ok with it, I would like to include it in dotnet/performance repo, just as we are planning to do (from long time, sigh!) for ImageSharp benchmark in dotnet/performance#2149. That way, we can track the performance in our daily builds.

from bepuphysics2.

Alright, there's now a selection of benchmarks: https://github.com/bepu/bepuphysics2/tree/master/DemoBenchmarks

I haven't yet included microbenchmarks of the Tree- I'll probably do that after I get done with its rewrite.

It uses project references as-is, but the prerelease versions of the nuget package should work too.

If you are ok with it, I would like to include it in dotnet/performance repo, just as we are planning to do (from long time, sigh!) for ImageSharp benchmark in dotnet/performance#2149. That way, we can track the performance in our daily builds.

Sounds good; you have my permission!

from bepuphysics2.

Sweet. Thanks for the quick turnaround. Is there an easy way to tell which of the 3 categories they belong to? Perhaps a README.md in that folder? I will start the process to port them over.

from bepuphysics2.

I did a run of these benchmarks (results dotnet/performance#2672 (comment)) and seems like it takes around 45 minutes to complete. Is it possible to trim them down little bit?

from bepuphysics2.

Yup, some have obviously higher value than others- about what time budget are you targeting? I'll see if I can prioritize within that.

from bepuphysics2.

We can target for 15 minutes and see how many we can accomodate. We can adjust it depending on the coverage we get in those 15 minutes.

from bepuphysics2.

d9b1eea brings it to around 12 minutes on my machine. Some benchmarks are merged, others are punted to 'Deep' classes to make it easier to skip. Readme's updated with details; the full suite is still there optionally.

from bepuphysics2.

I tried it on my machine, and I still see it for 40 minutes. Do you see any benchmarks in this list that should have not run?

Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=7.0.100-preview.6.22352.1
  [Host]     : .NET 7.0.0 (7.0.22.32404), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.0 (7.0.22.32404), X64 RyuJIT AVX2
  Job-NGOYYI : .NET 7.0.0 (7.0.22.32404), X64 RyuJIT AVX2

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  Median : Value separating the higher half of all measurements (50th percentile)
  1 ns   : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
Global total time: 00:40:22 (2422.3 sec), executed benchmarks: 128
// * Artifacts cleanup *

from bepuphysics2.

Hey, what is the full command for running benchmarks with filters from the command line?

Somehow, this isn't working for me and this is the error.
dotnet run -c Release --filter *CollisionBatcherTaskBenchmarks*

from bepuphysics2.