datastax-labs / hunter Goto Github PK

Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the --magnitude argument filters across all metrics.

One option could be to allow setting multiple metric=min_magnitude pairs with the --magnitude flag, or exclusively a single float. This would be pretty verbose... Ex:

hunter ... --magnitude p50=0.05 --magnitude p99=0.2

Another option would be setting the min_magnitude in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.

templates:
  common_metrics:
    metrics:
      p50:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.05
      p99:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.2
tests:
  local2:
    type: csv
    file: tests/resources/sample.csv
    time_column: time
    metrics:
      m1:
        column: metric1
        direction: 1
      m2:
        column: metric2
        direction: -1
        min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
    attributes: [ commit ]

  remote1:
    inherit: [common_metrics] # min_magnitude set via inheritance
    prefix: "performance_regressions.my_product.%{BRANCH}.test1"

Hunter has a lot of potential, why has progress halted for so long?

On macOS running on Apple Silicon, version 1.19.0 of numpy no longer works as it is too old. More information can be found here on this issue in the numpy repo. I went through this problem and couldn't install hunter due to it so I tried upgrading numpy and things work with tests passing so I propose upgrading numpy to version 1.26.1 and python to ">=3.9,<3.13" to support the change.