Code Monkey home page Code Monkey logo

benchdiff's Introduction

benchdiff

Go Reference ci

Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show the delta.

What it does

Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference.

It essentially combines multiple git, go test -bench and benchstat commands into one.

These are the basic steps benchdiff performs:

  • Runs go test -bench ... on your current worktree and saves the results to a cache directory.
  • Creates a new worktree at the base ref
  • Runs go test -bench ... on the base worktree and saves the results to cache again.
  • Runs benchstat to compare the base and head results from cache.

GitHub Action

benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action

Usage

benchdiff --help output

Usage: benchdiff

benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.

More documentation at https://github.com/willabides/benchdiff.

Flags:
  -h, --help       Show context-sensitive help.
      --version    Output the benchdiff version and exit.
      --debug      write verbose output to stderr

  --base-ref="HEAD"    The git ref to be used as a baseline.
  --cooldown=100ms     How long to pause for cooldown between head and base runs.
  --force-base         Rerun benchmarks on the base reference even if the output already exists.
  --git-cmd="git"      The executable to use for git commands.
  --json               Format output as JSON.
  --on-degrade=0       Exit code when there is a statistically significant degradation in the
                       results.
  --tolerance=10.0     The minimum percent change before a result is considered degraded.

benchmark command line
  --bench="."              Run only those benchmarks matching a regular expression. To run all
                           benchmarks, use '--bench .'.
  --benchmark-args=args    Override the default args to the go command. This may be a template.
                           See https://github.com/willabides/benchdiff for details."
  --benchmark-cmd="go"     The command to use for benchmarks.
  --benchmem               Memory allocation statistics for benchmarks.
  --benchtime=STRING       Run enough iterations of each benchmark to take t, specified as a
                           time.Duration (for example, --benchtime 1h30s). The default is 1 second
                           (1s). The special syntax Nx means to run the benchmark N times (for
                           example, -benchtime 100x).
  --count=10               Run each benchmark n times. If --cpu is set, run n times for each
                           GOMAXPROCS value.'
  --cpu=GOMAXPROCS,...     Specify a list of GOMAXPROCS values for which the benchmarks should be
                           executed. The default is the current value of GOMAXPROCS.
  --packages="./..."       Run benchmarks in these packages.
  --show-bench-cmdline     Instead of running benchmarks, output the command that would be used and
                           exit.
  --tags=STRING            Set the -tags flag on the go test command
  --warmup-count=INT       Run benchmarks with -count=n as a warmup
  --warmup-time=STRING     When warmups are run, set -benchtime=n

benchstat options
  --alpha=0.05                 consider change significant if p < α
  --benchstat-output="text"    format for benchstat output (csv,html,markdown or text)
  --delta-test="utest"         significance test to apply to delta: utest, ttest, or none
  --geomean                    print the geometric mean of each file
  --norange                    suppress range columns (CSV and markdown only)
  --reverse-sort               reverse sort order
  --sort="none"                sort by order: delta, name, none
  --split="pkg,goos,goarch"    split benchmarks by labels

benchmark result cache
  --cache-dir=STRING    Override the default directory where benchmark output is kept.
  --clear-cache         Remove benchdiff files from the cache dir.
  --show-cache-dir      Output the cache dir and exit.

--benchmark-args

The default should work for most cases.

--benchmark-args is everything after "go" in the go test command that runs the benchmarks. You can use either pass in the raw values or use a go template.

Default value:

test {{ .Packages }} -run '^$'
{{- if .Bench }} -bench {{ .Bench }}{{end}}
{{- if .Count }} -count {{ .Count }}{{end}}
{{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}}
{{- if .CPU }} -cpu {{ .CPU }}{{ end }}
{{- if .Tags }} -tags "{{ .Tags }}"{{ end }}
{{- if .Benchmem }} -benchmem{{ end }}

Install

go get

go get -u github.com/willabides/benchdiff/cmd/benchdiff

bindown

Add a bindown dependency:

$ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml
$ bindown dependency add benchdiff benchdiff#benchdiff
Please enter a value for required variable "version":	<latest version>

benchdiff's People

Contributors

filosottile avatar willabides avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

benchdiff's Issues

Interrupting benchmark run persists partial cache file

I stated a benchdiff run, and then realized I had gotten it wrong and stopped it with ^C.

When I reran it, it said "skipping benchmark for ref "HEAD^" because output file exists" although the file in the cache was just a couple lines from the first benchmark.

Instead, the output should go to a temporary file and only get moved to the cache upon success.

type mismatch error in helpprinter when getting benchdiff

Hi,

I am getting following errors when I am trying to go get benchdiff.

❯ go get -u github.com/willabides/benchdiff/cmd/benchdiff
# github.com/willabides/benchdiff/cmd/benchdiff/internal/helpprinter
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:31:22: cannot use flag.Group (type *kong.Group) as type string in map index
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:33:14: cannot use flag.Group (type *kong.Group) as type string in map index
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:37:20: cannot use flags[i].Group (type *kong.Group) as type string in map index
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:37:49: cannot use flags[j].Group (type *kong.Group) as type string in map index
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:139:39: invalid operation: group[0].Group != "" (mismatched types *kong.Group and string)
../../../../go/pkg/mod/github.com/willabides/[email protected]/cmd/benchdiff/internal/helpprinter/helpprinter.go:140:35: invalid operation: group[0].Group + "GroupHelp" (mismatched types *kong.Group and string)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.