Provide a script to locally benchmark against the baseline about perf-ninja HOT 8 CLOSED

Thanks for the link. I haven't seen this project before. It looks great BTW. For now, if I need to benchmark something multiple times on Linux from the command line I simply use perf -stat -r 10 -- <./app>. But hyperfine may be slightly better way to do it.
For perf-ninja, all the labs are built on top of google benchmark library, which does all the measurements reporting. So hyperfine probably won't help us here.
What we need to do I think is something like this:

// for the current lab
- stash the changes
- rebuild & rerun the baseline lab
- restore the changes
- rebuild & rerun the modified lab
- measure the speedup.

from perf-ninja.

Actually, there is a potential solution, which doesn't require changes:

# 1. Build baseline
# 2. Cache baseline results
$ ./lab --benchmark_out_format=json --benchmark_out=result_baseline.json
# 3. Make changes to the lab & rebuild
$ ./lab --benchmark_out_format=json --benchmark_out=result_opt.json
# 4. Compare results:
$ compare.py benchmarks result_baseline.json result_opt.json

compare.py script is here.

@mark2185, can you please check if this would work for you?

from perf-ninja.

I haven't nearly dug enough into this wonderful project (yet) to know the details of how you are doing this sort of thing, but I thought I should mention hyperfine as a possible tool to use for this if you aren't already :) If I'm not mistaken, it was pretty much created specifically for use cases like this.

from perf-ninja.

The baseline lab doesn't have to be run so often since it rarely changes.

I think a script for running all benchmarks and caching the results would suffice, which would be used manually on every master update. Then you only need a second script to run for the 'current' lab and compare it to the cache.

from perf-ninja.

Yep, that should also work. In fact, we were talking about using this approach in CI with @andrewevstyukhin. Caching results in a JSON file is super easy with python. Does anyone want to help implementing it?

from perf-ninja.

Ah, yeah, I didn't know about the benchmark library, but that does look like it's probably the better choice here 👍 On the topic of Python scripts, there is apparently a small project named chronologger which runs benchmarks comparing git commits. Maybe that could be of some use as a reference?

from perf-ninja.

Fixed and described here:
https://github.com/dendibakh/perf-ninja/blob/main/GetStarted.md#local-experiments

from perf-ninja.

@mark2185, can you please check if this would work for you?

Yeah, sorry, I kept postponing this, but I'm sure it'll work out! Thanks! :)

from perf-ninja.