ichard26 / blackbench Goto Github PK

At the moment of writing doesn't come with much benchmark data to run black against. It's basically a 21.6b0 checkout of black's source code + one string micro target (based off of Jelle's discovery of ESP causing slowdowns). I'd appreciate more code files to throw to black.

I'm looking for two kinds of targets: "normal" and "micro" . Normal targets should represent real-world code (so the benchmarking data actually represents real-world performance). Micro targets should be small and are focused on one area of black formatting (and mostly exist to measure performance in a specific area, like string processing).

In terms of hard rules: normal targets shouldn't be bigger than ~2000 lines (this is to keep time requirements to run the benchmark based off the target managable), and micro targets shouldn't be bigger than ~400 lines.

Thank you in advance!

Soo I tried implementing this but I ended up breaking a few integration tests because the test suite is *that* brittle (or strict but that feels too positive for my questionable test suite :P) :/ I'mma defer this to the alpha where hopefully I find the time and energy to rewrite the integration tests to be less brittle (and just generally more maintainable).

Anyway in terms of implementation I'd like to see the following information be injected:

• Blackbench version;
• Black version;
• Task name & description; and
• Target name & description.

You can configure the benchmarking parameters directly through blackbench run out.json -- $pyperf-args. Which is quite useful to increase or debug benchmark stability. Unfortunately messing up the pyperf arguments is quite easy and leads to every single benchmark to fail ... let's catch this before any benchmark runs. Same thing with the black.Mode() configuration via --format-config.

• black.Mode(...)
• pyperf arguments
• black version & selected task

Also, if possible, try to at least warn if the current Black version is incompatible with the current task. Unfortunately development versions makes this difficult, so it's not too important if this gets deferred or ends up never implemented.

While I'm not ready to release blackbench to the greater world out there, I am getting close so it would be nice to have the infrastructure to do so in place by then. I know that since this project is using flit it's probably quite simple, but I'd like this project to be easy to maintain down the road (especially if I end up disappearing and someone else is now in charge).