test fixtures about benchmark HOT 5 CLOSED

This would have to fundamentally change the API I think. Currently, we just use macros to register user-defined static methods. With fixtures, we may have to change to the googletest model of defining classes from benchmarks.

Unless... Perhaps a 'fixture' can inherit from benchmark::State and be passed to the benchmark instead of the state. Ie, you'd have something like:

static void BM_Basic(benchmark::State& state) {
  while (state.KeepRunning()) {
    ...
  }
}
BENCHMARK(BM_basic);

class MyState : public benchmark::State {
 protected:
  MyState() : _count(0) {}
  ~MyState() final {
    std::stringstream ss;
    ss << _count;
    SetLabel(ss.str());
  }

  int _count;
};

static void BM_Fixture(MyState& state) {
  while (state.KeepRunning()) {
    ...
    ++_count;
  }
}

BENCHMARK_F(MyState, BM_Fixture);

We still need a second version of all the top-level registration macros that take a fixture, and we'd want to check that the given fixture is-a state. This will also require more templating on State...

What do you think, @EricWF?

from benchmark.

That makes sense to me for the most part. I agree that we shouldn't need to break the current API.

However I don't know if we should use the constructor/destructer for initialization/destruction. Since the fixture classes will likely have a static lifetime. I also don't know how your given example of BM_Fixture would have _count in scope.

from benchmark.

_count isn't in scope - that was a bug :P

Why would the fixture classes have static lifetime? RunInThread would need to create the state on the heap as a pointer using the correct fixture type (instead of on the stack as it is now) but it would have the same per-run lifetime. no?

from benchmark.

Hmm, Perhaps I'm not understanding what your fully suggesting. Could you explain further how you see these fixture classes working?

from benchmark.

I've tooled around and can't find a syntax i like. I was hoping to have either State be subclassed (but registering and instantiating that correctly is hard) so then I tried to have State contain an 'inner state' of some user-defined type, but that was equally disruptive.

It's possible now for users to create an object with static lifetime and call methods on it outside the KeepRunning loop so it's not a very high priority to implement.

My ideal would be that the user defines a type that they pass in through registration and an instance of that type lives for the lifetime of a benchmark run.

from benchmark.