Connect experimental/alp autotuning with harness' tuning hooks about iree-llvm-sandbox HOT 5 OPEN

@giuseros Do you think you'd be interested in merging the efforts here to have a common OpenTuner-tunable infrastructure? Are there any other blockers apart from #93, #94, #95 that would prevent things from connecting?

from iree-llvm-sandbox.

Hi @shabalind ,
Thanks for the input!

I don't think #93, #94, #95 are blockers, because those are only needed for us to have an easier life when analyzing performance (assembly, intermediate IRs, etc...). OpenTuner is agnostic of how we collect performance. Basically, here:

    compile_result = self.call_program(' '.join(cmd))
    run_cmd = './exec_matmul'
    run_result = self.call_program(run_cmd, limit=0.7)
    secs, flops = parse(run_result['stderr'])
    return Result(time=1/flops)

We compile, run the and feed the result back to the framework. Compile and run can be obtained in any way, as long as we are able to "extract" flops or time from the execution to implement the feedback loop.

If you could show me how to do that with the current python harness, then I can come up with a very simply autotuner example. Then it's mostly about how to structure the folder within the repo.

Thanks,
Giuseppe

from iree-llvm-sandbox.

@giuseros If you look at examples/matmul/bench.py you'll see a few key things that you need to build an equivalent of your snippet.

test_harness is roughly equivalent of call_program in your example. It evaluates a concrete configuration (in the same python process) and prints the performance results to stdout. Currently the function doesn't actually return anything yet, but it's easy to modify it to return performance numbers as well (PR to add this: #120 ).

The configuraton is defined by an expert, which is a composition of a number of transformations (see experts.py). Each transformation can be parametrized by variables (see transforms.py). Variables are roughly equivalent to OpenTuner parameters: they are typed and have a clear precise domain of all possible valid values.

Harness accepts an instantiation of expert to some concrete values and runs it on a given problem definition (i.e. matmul vs conv vs reduction vs ...) + sizes + information of wheather sizes should be passed statically or dynamically.

Once harness can be invoked programmatically to obtain performance numbers it looks like it should be easy to make an equivalent of the code snippet you provided above.

from iree-llvm-sandbox.

Hi @shabalind ,
It really seems easy now, thanks! I will have a stab a this in the next few days and cc you in the review

Thanks,
Giuseppe

from iree-llvm-sandbox.

@giuseros Great! Let me know if there is anything else that blocks your progress -- I'd be happy to help to push this forward.

from iree-llvm-sandbox.