Trying to get to the bottom of ginkgo vs go test performance.

go run . will run a few different scenarios and save the test run timings in test_runs. If a scenario is already saved in test_runs it is skipped. go run . force will forcibly rerun all the scenarios regardless of what is stored in test_runs

Either command spits out a summary of the various scenarios.

python visualize.py generates a plot

There are four scenarios, each runs the tests stored in sample:

• go test: simply runs go test sample
• go test compile: mimics Ginkgo's behavior of compiling the test first (go test -c -output="test") then running the test (./test).
• ginkgo_cli_base: runs the default Ginkgo cli, which first compiles the test and stores it in a temp directory then runs the test.
• ginkgo_cli_no_tmp: runs a modified Ginkgo cli, which first compiles the test and stores it in the package directory then runs the test.

For each scenario (where possible) we capture the Compile Time, Run Time, and Total Time. The following are stats from a 100-iteration run on my macbook pro:

And here's the plot:

• go test is 100ms faster than go test compile
• ginkgo_cli_no_tmp is equivalent to go test compile
• ginkgo_cli_base - i.e. the default ginkgo behavior that compiles the test into a tmp directory - sees notable latency in the Run Time - of about 500ms on average with a wide variance.

So. Apparently. Don't try to run binaries from a temp directory (at least on MacOS) as that introduces latency with a long-tail distribution (see the plot).

This performance exploration started - in part - due to this issue. On my laptop vanilla ginkgo takes ~4.1s but the no-temp-dir flavor of ginkgo takes ~3.3s.

Running ginkgo's unit tests shows a reduction in run time from ~16.3s to ~12.7s.