This fork includes the following CI configs files:
.github/workflows/ci.yml
.circleci/config.yml
.travis.yml
.gitlab-ci.yml
It was used to test these CI services. Results are published at earthly.dev/blog/ci-comparison/
The benchmarks follow the criteria:
-
They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
-
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
-
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.
My other benchmarks: jit-benchmarks, crystal-benchmarks-game
The measured values are:
- time spent for the benchmark execution (loading required data and code self-testing are not measured);
- memory consumption of the benchmark process, reported as
base
+increase
, wherebase
is the RSS before the benchmark andincrease
is the peak increase of the RSS during the benchmark; - energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM.
All values are presented as: median
±median absolute deviation
.
UPDATE: 2022-02-18
Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:
- Verbose (default). Prints the output immediately.
- Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
Racket (Syntax Objects) | 0.848±0.013 | 112.71±00.74 + 0.00±00.00 | 18.86±01.01 |
C++/g++ | 1.143±0.063 | 1.68±00.00 + 0.00±00.00 | 20.79±02.04 |
C/gcc | 1.319±0.006 | 0.73±00.00 + 0.00±00.00 | 28.21±00.90 |
Go | 1.367±0.044 | 3.35±00.05 + 0.00±00.00 | 26.07±02.29 |
C++/clang++ | 1.407±0.054 | 1.53±00.03 + 0.00±00.00 | 28.82±02.69 |
D/ldc2 | 1.420±0.018 | 3.14±00.06 + 0.00±00.00 | 29.74±00.87 |
Rust | 1.444±0.021 | 2.16±00.04 + 0.00±00.00 | 24.06±00.58 |
Vala/gcc | 1.453±0.021 | 4.33±00.03 + 0.00±00.00 | 27.56±01.83 |
D/gdc | 1.470±0.031 | 6.86±00.03 + 0.00±00.00 | 31.01±00.88 |
Nim/gcc | 1.489±0.052 | 1.92±00.09 + 0.00±00.00 | 28.67±02.70 |
C/clang | 1.526±0.018 | 0.68±00.00 + 0.00±00.00 | 26.22±00.52 |
Vala/clang | 1.581±0.014 | 4.37±00.03 + 0.00±00.00 | 30.19±00.52 |
Java | 1.648±0.017 | 37.17±00.23 + 1.56±00.36 | 36.73±00.64 |
Go/gccgo | 1.678±0.094 | 22.37±00.12 + 0.00±00.00 | 33.14±01.98 |
Zig | 1.682±0.048 | 0.72±00.00 + 0.00±00.00 | 33.60±03.41 |
Nim/clang | 1.768±0.065 | 2.38±00.04 + 0.00±00.00 | 31.90±02.28 |
Kotlin/JVM | 1.797±0.036 | 40.57±00.29 + 1.19±00.60 | 33.54±01.29 |
OCaml | 1.956±0.033 | 2.85±00.05 + 2.27±00.03 | 42.49±02.50 |
MLton | 2.039±0.044 | 1.47±00.07 + 0.25±00.00 | 44.69±02.30 |
Racket | 2.131±0.137 | 95.76±00.26 + 0.00±00.00 | 42.11±03.49 |
C#/.NET Core | 2.256±0.063 | 34.20±00.05 + 0.00±00.00 | 43.70±03.79 |
V/clang | 2.288±0.096 | 0.74±00.00 + 0.00±00.00 | 49.66±02.87 |
Julia | 2.392±0.097 | 207.12±00.32 + 0.70±00.03 | 35.68±01.69 |
Crystal | 2.433±0.068 | 3.52±00.01 + 0.00±00.00 | 43.41±02.12 |
F#/.NET Core | 2.551±0.068 | 37.80±00.08 + 0.61±00.00 | 45.74±02.08 |
C#/Mono | 2.803±0.104 | 20.40±00.03 + 0.00±00.00 | 57.16±05.19 |
Scala | 2.988±0.055 | 75.72±00.09 + 207.00±30.15 | 60.22±04.14 |
Chez Scheme | 3.022±0.122 | 24.83±00.04 + 4.43±00.08 | 55.62±02.53 |
D/dmd | 3.731±0.086 | 3.76±00.04 + 0.00±00.00 | 65.93±01.93 |
Node.js | 4.569±0.094 | 35.26±00.04 + 1.10±00.09 | 78.53±01.73 |
Haskell (MArray) | 4.822±0.141 | 3.85±00.07 + 1.11±00.00 | 87.24±02.20 |
Swift | 5.678±0.081 | 13.17±00.06 + 0.00±00.00 | 128.69±02.71 |
Lua/luajit | 7.852±0.387 | 2.42±00.03 + 0.00±00.00 | 142.46±12.07 |
Ruby/truffleruby (--jvm) | 7.983±0.381 | 338.00±02.50 + 562.25±53.63 | 282.72±06.68 |
Ruby/truffleruby | 9.986±0.467 | 278.40±01.31 + 707.98±60.03 | 264.21±20.52 |
Python/pypy | 13.383±0.487 | 63.75±00.14 + 46.92±00.05 | 273.75±15.52 |
Haskell | 14.413±0.224 | 3.95±00.12 + 0.82±00.17 | 343.68±08.44 |
Ruby (--jit) | 42.420±1.098 | 271.26±00.04 + 0.02±00.00 | 759.89±21.19 |
Lua | 53.184±1.056 | 2.28±00.03 + 0.00±00.00 | 929.47±10.98 |
Elixir | 59.667±1.157 | 76.02±01.32 + 0.00±00.00 | 1077.70±71.88 |
Ruby | 91.057±2.093 | 14.72±00.03 + 0.00±00.00 | 1799.94±149.19 |
Ruby/jruby | 108.798±1.292 | 185.06±01.75 + 145.62±31.24 | 2173.96±119.87 |
Python | 166.013±6.278 | 10.45±00.01 + 0.00±00.00 | 3395.94±151.28 |
Tcl (FP) | 238.309±4.381 | 4.31±00.05 + 0.00±00.00 | 5134.52±121.99 |
Perl | 319.914±2.781 | 6.59±00.04 + 0.00±00.00 | 5786.86±30.09 |
Tcl (OOP) | 481.582±5.443 | 4.35±00.06 + 0.00±00.00 | 8798.96±234.20 |
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
C++/g++ | 10.950±0.251 | 3.48±00.06 + 0.52±00.00 | 218.96±08.02 |
C/gcc | 12.523±0.183 | 0.71±00.03 + 1.01±00.02 | 257.16±05.69 |
V/gcc | 13.707±0.426 | 1.79±00.04 + 1.03±00.00 | 285.62±08.07 |
Rust | 13.731±0.171 | 2.12±00.07 + 0.28±00.03 | 235.48±05.28 |
C++/clang++ | 14.174±0.325 | 1.52±00.03 + 2.12±00.02 | 299.19±09.15 |
Vala/gcc | 14.663±0.363 | 4.18±00.03 + 1.86±00.08 | 291.60±26.98 |
D/ldc2 | 14.976±0.486 | 3.25±00.04 + 0.77±00.00 | 283.84±27.12 |
C/clang | 15.470±0.404 | 0.73±00.00 + 0.98±00.01 | 323.58±06.07 |
D/gdc | 16.506±0.701 | 6.85±00.04 + 0.56±00.04 | 337.21±15.95 |
Racket (Syntax Objects) | 16.950±0.862 | 112.16±00.33 + 68.29±00.23 | 361.35±27.86 |
Kotlin/JVM | 17.402±0.433 | 40.71±00.16 + 1.10±00.29 | 337.89±11.57 |
Go | 17.836±0.743 | 3.22±00.02 + 1.26±00.00 | 324.48±21.86 |
Nim/gcc | 18.680±0.826 | 1.97±00.03 + 0.51±00.00 | 372.68±27.56 |
Zig | 18.718±0.372 | 1.54±00.05 + 0.77±00.00 | 337.39±20.87 |
Vala/clang | 18.810±0.517 | 4.16±00.07 + 1.86±00.09 | 401.28±40.20 |
Crystal | 19.709±0.364 | 3.45±00.03 + 0.43±00.02 | 347.61±10.56 |
Scala | 20.175±0.496 | 77.13±00.16 + 126.70±00.21 | 413.26±22.62 |
Java | 20.628±0.997 | 37.38±00.53 + 1.93±00.45 | 424.03±32.83 |
Nim/clang | 20.684±0.372 | 2.44±00.05 + 0.54±00.03 | 432.57±15.62 |
Go/gccgo | 21.434±0.736 | 22.44±00.13 + 1.28±00.01 | 441.50±28.10 |
Swift | 21.689±0.758 | 13.93±00.17 + 0.00±00.00 | 475.67±23.43 |
V/clang | 22.521±0.469 | 1.76±00.03 + 0.77±00.00 | 466.48±25.32 |
C#/.NET Core | 23.313±0.767 | 33.81±00.10 + 1.14±00.00 | 475.38±43.97 |
OCaml | 34.780±1.291 | 4.10±00.04 + 8.86±01.05 | 683.02±52.59 |
Racket | 39.128±1.913 | 95.71±00.25 + 0.26±00.00 | 839.39±90.78 |
Chez Scheme | 42.220±1.691 | 25.69±00.06 + 3.68±00.02 | 813.73±36.13 |
D/dmd | 43.995±1.064 | 3.81±00.05 + 0.77±00.00 | 902.73±36.53 |
Lua/luajit | 44.276±1.444 | 2.43±00.04 + 0.44±00.00 | 936.01±28.17 |
F#/.NET Core | 44.921±1.647 | 37.84±00.07 + 2.15±00.00 | 872.03±63.91 |
C#/Mono | 45.631±0.283 | 20.30±00.07 + 0.89±00.00 | 795.13±10.84 |
MLton | 47.547±1.556 | 1.59±00.09 + 4.11±00.00 | 1030.05±69.26 |
Node.js | 50.279±1.020 | 35.20±00.05 + 5.20±00.00 | 942.47±48.01 |
Julia | 51.288±0.979 | 208.13±00.12 + 0.65±00.01 | 1124.20±40.28 |
Python/pypy | 63.640±1.713 | 63.78±00.14 + 47.77±00.01 | 1295.30±103.67 |
Haskell (MArray) | 69.589±0.591 | 3.79±00.08 + 2.63±00.03 | 1285.05±07.56 |
Ruby/truffleruby (--jvm) | 123.667±3.258 | 335.56±07.72 + 548.28±93.60 | 2637.61±76.37 |
Ruby/truffleruby | 137.962±3.223 | 280.17±03.67 + 714.62±46.79 | 2642.24±170.32 |
Haskell | 221.219±2.389 | 3.92±00.06 + 26.16±00.00 | 5035.28±152.83 |
Testing base64 encoding/decoding of the large blob into the newly allocated buffers.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
C/clang (aklomp) | 0.137±0.004 | 2.05±00.02 + 0.00±00.00 | 3.42±00.15 |
C/gcc (aklomp) | 0.140±0.006 | 2.03±00.02 + 0.00±00.00 | 3.54±00.10 |
Rust | 1.106±0.044 | 2.58±00.07 + 0.01±00.01 | 24.75±01.65 |
C/gcc | 1.207±0.053 | 2.01±00.02 + 0.00±00.00 | 23.13±01.89 |
C/clang | 1.226±0.045 | 2.00±00.02 + 0.00±00.00 | 24.68±02.34 |
V/clang | 1.272±0.035 | 1.93±00.04 + 0.00±00.00 | 26.59±02.32 |
Nim/gcc | 1.372±0.012 | 2.31±00.09 + 4.12±00.03 | 23.48±00.43 |
Nim/clang | 1.507±0.063 | 2.82±00.07 + 4.38±00.03 | 33.97±02.72 |
D/ldc2 | 1.867±0.024 | 3.47±00.02 + 3.63±00.00 | 33.14±00.30 |
Ruby (--jit) | 2.078±0.046 | 271.68±00.05 + 67.07±00.04 | 46.75±03.33 |
Ruby | 2.099±0.033 | 15.13±00.05 + 58.73±00.00 | 46.06±01.90 |
Vala/clang | 2.187±0.023 | 5.54±00.03 + 0.44±00.04 | 44.67±01.14 |
Java | 2.205±0.062 | 38.96±00.16 + 262.42±04.83 | 41.97±01.95 |
Vala/gcc | 2.361±0.103 | 5.58±00.05 + 0.46±00.03 | 54.23±04.60 |
C++/clang++ (libcrypto) | 2.370±0.072 | 5.12±00.03 + 0.08±00.00 | 52.52±03.27 |
C++/g++ (libcrypto) | 2.391±0.115 | 5.72±00.04 + 0.08±00.00 | 51.61±04.25 |
Crystal | 2.419±0.010 | 3.92±00.03 + 1.17±00.01 | 48.52±00.35 |
Scala | 2.435±0.023 | 74.49±00.26 + 240.30±16.40 | 46.46±01.14 |
Kotlin/JVM | 2.451±0.048 | 41.52±00.68 + 241.74±00.39 | 57.94±02.46 |
Go | 2.547±0.007 | 4.40±00.04 + 5.56±00.23 | 50.55±00.71 |
Perl (MIME::Base64) | 2.825±0.094 | 14.33±00.04 + 0.07±00.06 | 55.10±04.96 |
PHP | 2.958±0.047 | 16.74±00.09 + 0.00±00.00 | 63.91±04.43 |
Node.js | 3.058±0.046 | 34.98±00.05 + 36.73±00.09 | 59.94±01.24 |
Go/gccgo | 3.176±0.009 | 23.43±00.19 + 8.72±00.28 | 64.46±00.47 |
D/gdc | 3.455±0.130 | 7.12±00.05 + 3.41±00.01 | 66.42±06.31 |
D/dmd | 3.706±0.078 | 3.94±00.03 + 3.62±00.00 | 70.75±03.68 |
Python | 4.077±0.137 | 10.16±00.04 + 0.18±00.00 | 79.47±08.79 |
Zig | 4.106±0.040 | 1.50±00.02 + 0.37±00.00 | 93.20±04.91 |
Python/pypy | 4.727±0.123 | 63.59±00.10 + 47.59±00.08 | 106.54±05.62 |
Julia | 4.848±0.060 | 226.41±00.43 + 52.16±09.93 | 113.62±03.55 |
F#/.NET Core | 5.150±0.021 | 38.55±00.03 + 39.05±06.54 | 88.05±00.95 |
C#/.NET Core | 5.230±0.038 | 34.58±00.05 + 32.37±03.19 | 89.02±01.14 |
Tcl | 5.931±0.092 | 4.88±00.02 + 0.18±00.03 | 104.96±02.94 |
Ruby/truffleruby (--jvm) | 6.189±0.098 | 343.99±06.43 + 262.94±42.01 | 133.48±06.65 |
C#/Mono | 7.444±0.281 | 20.87±00.06 + 18.47±00.05 | 168.84±12.40 |
Ruby/jruby | 7.554±0.145 | 183.80±02.99 + 176.76±03.83 | 160.35±09.12 |
Perl (MIME::Base64::Perl) | 14.333±0.442 | 15.62±00.05 + 0.27±00.04 | 303.59±08.81 |
Ruby/truffleruby | 18.930±0.680 | 280.37±02.60 + 285.21±20.17 | 366.82±27.92 |
Testing parsing and simple calculating of values from a big JSON file.
Few notes:
- gason mutates input strings;
- simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
- DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
- DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
C++/g++ (simdjson On-Demand) | 0.090±0.001 | 113.41±00.06 + 59.81±00.00 | 1.93±00.08 |
C++/clang++ (simdjson On-Demand) | 0.095±0.002 | 113.07±00.06 + 59.81±00.00 | 2.11±00.18 |
C++/g++ (DAW JSON Link NoCheck) | 0.096±0.003 | 113.18±00.05 + 0.00±00.00 | 2.02±00.11 |
C++/clang++ (DAW JSON Link NoCheck) | 0.103±0.005 | 112.49±00.02 + 0.00±00.00 | 2.21±00.25 |
C++/g++ (DAW JSON Link) | 0.104±0.003 | 113.25±00.03 + 0.00±00.00 | 2.65±00.18 |
C++/clang++ (DAW JSON Link) | 0.123±0.002 | 112.52±00.10 + 0.00±00.00 | 2.56±00.27 |
C++/clang++ (simdjson DOM) | 0.135±0.005 | 113.03±00.07 + 176.60±00.26 | 3.19±00.20 |
C++/g++ (simdjson DOM) | 0.136±0.003 | 113.44±00.01 + 176.60±00.00 | 3.01±00.10 |
Rust (Serde Custom) | 0.144±0.004 | 112.00±00.07 + 0.00±00.00 | 2.97±00.26 |
Rust (Serde Typed) | 0.147±0.005 | 111.93±00.09 + 11.67±00.13 | 3.14±00.34 |
C++/g++ (gason) | 0.165±0.006 | 113.15±00.06 + 96.80±00.03 | 3.27±00.20 |
C++/clang++ (gason) | 0.191±0.002 | 112.49±00.05 + 96.97±00.00 | 3.69±00.09 |
C++/g++ (RapidJSON) | 0.213±0.003 | 113.23±00.05 + 128.82±00.02 | 4.11±00.24 |
D/ldc2 (Mir Asdf DOM) | 0.247±0.008 | 112.83±00.04 + 61.35±00.00 | 5.01±00.42 |
D/ldc2 (Mir Amazon's Ion DOM) | 0.258±0.008 | 112.79±00.06 + 16.25±00.00 | 4.96±00.43 |
C++/clang++ (RapidJSON) | 0.260±0.009 | 112.53±00.09 + 129.00±00.01 | 5.90±00.40 |
C++/g++ (RapidJSON Precise) | 0.270±0.005 | 113.25±00.04 + 128.85±00.04 | 6.17±00.23 |
C++/clang++ (RapidJSON Precise) | 0.417±0.009 | 112.58±00.06 + 129.00±00.00 | 7.63±00.24 |
C++/g++ (RapidJSON SAX) | 0.473±0.021 | 112.93±00.01 + 0.00±00.00 | 10.24±00.69 |
C++/g++ (Boost.JSON) | 0.500±0.016 | 113.21±00.04 + 435.85±00.03 | 12.05±00.83 |
Nim/clang (jsony) | 0.533±0.014 | 112.14±00.02 + 42.31±00.03 | 9.99±00.44 |
C++/clang++ (RapidJSON SAX) | 0.539±0.014 | 194.79±00.07 + 0.00±00.00 | 9.65±00.71 |
Nim/gcc (jsony) | 0.556±0.029 | 111.61±00.05 + 42.34±00.06 | 12.71±00.92 |
C++/g++ (RapidJSON SAX Precise) | 0.559±0.015 | 113.00±00.01 + 0.00±00.00 | 9.78±00.23 |
C++/clang++ (Boost.JSON) | 0.565±0.019 | 112.55±00.04 + 436.25±00.00 | 11.17±00.75 |
Node.js | 0.650±0.026 | 145.06±00.07 + 184.66±00.33 | 16.07±00.95 |
Go (jsoniter) | 0.657±0.016 | 231.09±00.09 + 1.43±00.09 | 13.75±00.74 |
C++/clang++ (RapidJSON SAX Precise) | 0.665±0.029 | 194.77±00.04 + 0.00±00.00 | 11.86±00.41 |
Java (DSL-JSON) | 0.737±0.010 | 263.00±00.49 + 210.85±11.03 | 18.70±00.72 |
Python/pypy | 0.780±0.032 | 283.42±00.15 + 123.29±00.00 | 16.10±02.00 |
Rust (Serde Untyped) | 0.831±0.040 | 111.96±00.06 + 839.98±00.00 | 17.55±01.77 |
V/gcc | 0.844±0.029 | 111.24±00.02 + 496.21±00.06 | 19.19±01.14 |
V/clang | 0.884±0.020 | 111.23±00.03 + 496.21±00.06 | 17.92±01.90 |
Zig | 0.924±0.038 | 110.80±00.02 + 12.18±00.00 | 19.69±01.71 |
Julia (JSON3) | 0.924±0.032 | 462.42±00.29 + 257.55±00.49 | 17.61±00.72 |
C#/.NET Core (System.Text.Json) | 0.985±0.030 | 478.82±00.04 + 138.78±00.00 | 18.62±01.95 |
Nim/clang (Packedjson) | 0.999±0.045 | 112.74±00.06 + 293.91±00.00 | 22.41±01.07 |
Perl (Cpanel::JSON::XS) | 1.004±0.038 | 124.90±00.03 + 402.77±00.00 | 22.90±01.36 |
Crystal (Pull) | 1.035±0.029 | 113.89±00.05 + 18.18±00.02 | 20.58±00.30 |
Go | 1.083±0.047 | 117.23±00.09 + 83.28±00.25 | 20.20±01.63 |
Crystal (Schema) | 1.084±0.047 | 113.86±00.08 + 48.61±00.00 | 20.99±01.20 |
Nim/gcc (Packedjson) | 1.137±0.035 | 112.18±00.08 + 293.91±00.00 | 25.53±01.02 |
PHP | 1.192±0.046 | 125.95±00.06 + 682.09±00.00 | 25.15±01.84 |
Crystal | 1.342±0.037 | 113.88±00.04 + 392.07±00.09 | 25.04±02.04 |
Nim/clang | 1.456±0.039 | 112.70±00.04 + 924.80±00.03 | 31.77±02.54 |
Clojure | 1.536±0.065 | 450.20±03.25 + 499.05±09.78 | 40.77±02.65 |
C++/g++ (json-c) | 1.557±0.052 | 113.30±00.12 + 1215.96±00.00 | 35.02±02.95 |
C++/clang++ (Nlohmann) | 1.602±0.055 | 112.66±00.05 + 360.15±00.03 | 36.01±03.19 |
C++/clang++ (json-c) | 1.617±0.021 | 112.70±00.01 + 1216.09±00.02 | 30.62±01.36 |
Nim/gcc | 1.642±0.030 | 112.18±00.09 + 919.39±00.03 | 30.55±01.62 |
C++/g++ (Nlohmann) | 1.680±0.077 | 113.21±00.12 + 447.96±00.08 | 36.12±01.71 |
Python | 1.703±0.054 | 120.14±00.02 + 377.33±00.00 | 39.86±01.82 |
Go/gccgo | 1.705±0.020 | 137.43±00.26 + 83.45±00.04 | 31.40±00.86 |
CPython (UltraJSON) | 1.724±0.021 | 121.89±00.03 + 547.29±01.16 | 37.53±01.69 |
C#/.NET Core | 1.749±0.034 | 486.57±00.21 + 294.18±00.01 | 31.46±01.37 |
Ruby (--jit) | 1.925±0.066 | 381.21±00.03 + 262.95±00.00 | 43.85±02.16 |
Ruby | 1.970±0.076 | 124.65±00.05 + 262.92±00.01 | 41.77±04.36 |
F#/.NET Core (System.Text.Json) | 2.348±0.099 | 486.03±00.07 + 452.20±01.80 | 45.45±04.39 |
C#/Mono | 2.377±0.153 | 476.10±00.10 + 0.19±00.01 | 47.15±04.55 |
D/ldc2 | 2.442±0.051 | 113.13±00.09 + 680.13±00.03 | 49.10±02.48 |
Scala (uPickle) | 2.569±0.085 | 303.83±00.12 + 714.35±50.07 | 53.86±02.62 |
Ruby (YAJL) | 2.629±0.043 | 124.57±00.03 + 279.79±00.00 | 49.00±02.29 |
D/gdc | 2.899±0.035 | 116.72±00.02 + 600.57±00.01 | 64.11±02.12 |
Haskell | 3.490±0.107 | 115.97±00.03 + 715.06±00.12 | 77.20±05.47 |
Rust (jq) | 3.699±0.127 | 113.83±00.01 + 778.38±00.26 | 80.25±02.99 |
C++/clang++ (Boost.PropertyTree) | 3.867±0.111 | 194.94±00.05 + 1232.86±00.03 | 79.39±05.54 |
Ruby/jruby | 3.867±0.072 | 459.34±07.57 + 1182.21±110.62 | 95.92±07.95 |
C++/g++ (Boost.PropertyTree) | 4.057±0.156 | 113.12±00.02 + 1440.14±00.03 | 82.94±07.71 |
Vala/clang | 5.012±0.164 | 114.73±00.03 + 932.45±00.02 | 97.95±08.40 |
Vala/gcc | 5.109±0.192 | 114.72±00.02 + 996.94±00.04 | 101.13±09.80 |
D/dmd | 5.449±0.092 | 113.78±00.02 + 680.12±00.03 | 99.31±06.83 |
Perl (JSON::Tiny) | 12.389±0.059 | 125.54±00.06 + 528.63±00.01 | 219.19±06.26 |
Ruby/truffleruby (--jvm) | 20.231±0.636 | 518.25±33.14 + 1531.78±51.59 | 658.38±20.97 |
Ruby/truffleruby | 31.058±1.111 | 432.55±05.15 + 2239.55±42.82 | 741.89±33.73 |
Testing allocating and multiplying matrices.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
D/ldc2 (lubeck) | 0.082±0.003 | 7.09±00.02 + 55.94±00.12 | 2.93±00.74 |
C++/g++ (Eigen) | 0.091±0.007 | 23.69±12.86 + 66.30±12.89 | 2.80±00.29 |
Python (NumPy) | 0.108±0.006 | 28.38±00.04 + 57.59±00.04 | 4.37±00.58 |
C++/clang++ (Eigen) | 0.118±0.016 | 15.25±10.81 + 75.25±10.75 | 3.49±00.41 |
Java (ND4J) | 0.118±0.008 | 137.43±02.38 + 90.75±00.03 | 3.93±00.56 |
Nim/clang (Arraymancer) | 0.125±0.021 | 6.58±00.08 + 55.68±00.31 | 4.04±00.40 |
Rust (ndarray) | 0.129±0.005 | 3.10±00.06 + 67.97±00.00 | 4.65±00.37 |
Nim/gcc (Arraymancer) | 0.140±0.007 | 5.70±00.11 + 55.84±00.47 | 4.42±00.80 |
Julia (threads: 8) | 0.154±0.015 | 249.46±00.31 + 43.40±00.07 | 5.32±00.65 |
Julia (threads: 1) | 0.206±0.004 | 249.49±00.27 + 41.87±00.03 | 4.01±00.11 |
Julia (no BLAS) | 1.274±0.020 | 223.95±00.24 + 51.44±00.03 | 29.94±03.39 |
D/ldc2 | 2.058±0.047 | 3.57±00.03 + 70.26±00.13 | 43.34±01.51 |
D/dmd | 2.201±0.030 | 3.85±00.14 + 70.27±00.13 | 46.81±01.61 |
D/gdc | 2.245±0.047 | 7.14±00.02 + 70.04±00.02 | 45.11±00.81 |
Nim/gcc | 3.420±0.045 | 2.70±00.04 + 65.87±00.13 | 77.88±02.61 |
C/gcc | 3.456±0.085 | 1.71±00.32 + 68.51±00.23 | 74.79±02.64 |
C/clang | 3.474±0.090 | 1.76±00.35 + 68.38±00.35 | 70.66±01.70 |
Nim/clang | 3.483±0.032 | 3.09±00.14 + 81.73±04.25 | 72.13±01.80 |
Rust | 3.515±0.042 | 2.71±00.17 + 68.32±00.25 | 73.23±01.49 |
Zig | 3.536±0.038 | 1.73±00.06 + 68.58±00.00 | 74.47±02.16 |
Vala/gcc | 3.594±0.063 | 5.80±00.09 + 68.32±00.00 | 61.46±01.34 |
Java | 3.607±0.030 | 38.92±00.12 + 80.58±00.29 | 73.25±00.95 |
Swift | 3.611±0.096 | 7.84±00.15 + 68.91±00.02 | 73.49±01.95 |
Go/gccgo | 3.645±0.080 | 22.57±00.15 + 73.47±00.14 | 78.64±02.77 |
Go | 3.661±0.090 | 3.54±00.06 + 73.31±00.26 | 83.72±02.83 |
Vala/clang | 3.711±0.091 | 5.79±00.06 + 68.32±00.00 | 59.58±03.05 |
V/clang | 3.726±0.071 | 2.31±00.02 + 68.84±00.00 | 84.50±02.34 |
Scala | 3.790±0.045 | 75.11±00.21 + 144.23±00.43 | 79.63±01.18 |
Crystal | 3.818±0.101 | 4.30±00.04 + 59.67±00.01 | 83.75±06.03 |
V/gcc | 3.856±0.057 | 1.94±00.06 + 68.84±00.00 | 77.54±04.86 |
Kotlin/JVM | 4.037±0.033 | 40.09±00.14 + 80.13±00.28 | 78.12±01.49 |
Node.js | 4.059±0.055 | 40.39±00.07 + 70.68±00.13 | 77.87±05.83 |
Python/pypy | 6.603±0.058 | 64.86±00.07 + 69.14±00.05 | 117.64±03.61 |
C#/.NET Core | 7.545±0.085 | 33.17±00.05 + 69.46±00.00 | 128.54±03.88 |
C#/Mono | 11.652±0.080 | 20.27±00.04 + 69.01±00.01 | 214.67±21.08 |
Ruby/truffleruby | 25.218±0.727 | 315.64±07.64 + 877.79±215.04 | 548.71±51.22 |
Ruby/truffleruby (--jvm) | 44.597±0.947 | 392.42±17.45 + 745.43±116.26 | 1007.27±37.75 |
Ruby (--jit) | 202.034±6.199 | 272.42±00.02 + 68.84±00.00 | 3820.61±263.34 |
Ruby | 222.467±6.667 | 15.83±00.06 + 68.58±00.00 | 4145.66±315.71 |
Python | 238.462±7.000 | 10.52±00.02 + 68.58±00.00 | 4496.58±343.38 |
Tcl | 321.774±7.723 | 7.21±00.01 + 400.44±00.00 | 6297.82±475.82 |
Perl | 402.410±4.590 | 9.04±00.03 + 599.72±00.08 | 8259.27±234.35 |
Ruby/jruby | 535.502±17.283 | 260.31±03.16 + 1172.21±31.18 | 10768.71±199.71 |
Testing:
- generating primes using the optimized sieve of Atkin;
- prefix search for their decimal numbers using Trie data structure.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
Zig | 0.074±0.001 | 1.42±00.02 + 55.56±02.12 | 1.59±00.07 |
Rust | 0.114±0.003 | 2.35±00.09 + 77.02±00.00 | 2.40±00.14 |
Java | 0.163±0.003 | 38.34±00.21 + 102.58±00.41 | 4.87±00.05 |
Crystal | 0.170±0.002 | 3.53±00.08 + 88.55±00.03 | 3.73±00.04 |
C++/clang++ | 0.174±0.002 | 3.16±00.05 + 75.41±00.01 | 3.79±00.20 |
C++/g++ | 0.184±0.008 | 3.52±00.03 + 84.80±00.00 | 3.64±00.26 |
V/clang | 0.187±0.003 | 1.72±00.04 + 270.91±03.22 | 3.60±00.11 |
V/gcc | 0.189±0.009 | 1.75±00.05 + 263.56±04.06 | 3.82±00.40 |
Node.js | 0.270±0.007 | 34.45±00.08 + 176.28±01.54 | 7.12±00.08 |
Lua/luajit | 0.404±0.015 | 2.56±00.06 + 156.05±00.57 | 8.21±00.68 |
Scala | 0.416±0.005 | 75.49±00.09 + 230.85±01.07 | 13.05±00.56 |
Python/pypy | 0.836±0.018 | 63.07±00.16 + 251.34±00.04 | 18.52±00.48 |
Lua | 1.656±0.021 | 2.27±00.04 + 284.17±00.48 | 34.45±00.71 |
Ruby (--jit) | 1.851±0.018 | 271.23±00.02 + 147.76±00.00 | 31.36±00.73 |
Ruby/truffleruby | 1.959±0.047 | 276.10±03.31 + 418.65±56.56 | 52.68±05.10 |
Ruby/truffleruby (--jvm) | 2.089±0.069 | 334.89±04.28 + 472.94±77.31 | 76.26±01.68 |
Ruby | 2.356±0.101 | 14.69±00.06 + 147.65±00.00 | 46.89±05.81 |
Ruby/jruby | 2.852±0.054 | 182.88±01.82 + 431.31±33.87 | 69.85±06.73 |
Python | 4.629±0.151 | 10.32±00.04 + 235.70±00.90 | 95.65±04.32 |
CPU: Intel(R) Core(TM) i7-10710U
Base Docker image: Debian GNU/Linux bookworm/sid
Language | Version |
---|---|
.NET Core | 6.0.102 |
C#/.NET Core | 4.0.1-1.22053.6 (9942dc95) |
C#/Mono | 6.12.0.122 |
Chez Scheme | 9.5.4 |
Clojure | "1.10.3" |
Crystal | 1.3.2 |
D/dmd | v2.098.1 |
D/gdc | 11.2.0 |
D/ldc2 | 1.28.1 |
Elixir | 1.12.2 |
F#/.NET Core | 12.0.0.0 for F# 6.0 |
Go | go1.17.7 |
Go/gccgo | 11.2.0 |
Haskell | 9.0.1 |
Java | 17.0.2 |
Julia | v"1.7.2" |
Kotlin | 1.6.10 |
Lua | 5.4.4 |
Lua/luajit | 2.1.0-beta3 |
MLton | 20210117 |
Nim | 1.6.4 |
Node.js | v17.5.0 |
OCaml | 4.13.1 |
PHP | 8.1.2 |
Perl | v5.34.0 |
Python | 3.9.10 |
Python/pypy | 7.3.7-final0 for Python 3.8.12 |
Racket | "8.4" |
Ruby | 3.1.0p0 |
Ruby/jruby | 9.3.3.0 |
Ruby/truffleruby | 22.0.0.2 |
Rust | 1.58.1 |
Scala | 3.1.1 |
Swift | 5.5.3 |
Tcl | 8.6 |
V | 0.2.4 efdbe9d |
Vala | 0.54.6 |
Zig | 0.9.0 |
clang/clang++ | 13.0.1 |
gcc/g++ | 11.2.0 |
Build the image:
$ docker build docker/ -t benchmarks
Run the image:
$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>
where <cmd>
is:
versions
(print installed language versions);shell
(start the shell);brainfuck bench
(build and run Brainfuck bench.b benchmarks);brainfuck mandel
(build and run Brainfuck mandel.b benchmarks);base64
(build and run Base64 benchmarks);json
(build and run Json benchmarks);matmul
(build and run Matmul benchmarks);primes
(build and run Primes benchmarks);
Please note that the actual measurements provided in the project are taken semi-manually (via shell
) as the full update takes days and could have occassional issues in Docker.
There is a Makefile
that could be used to simplify Docker usage:
make build
(build the image);make versions
(run the image with theversions
command);make shell
(run the image with the `shell' command);make toc
(utility rule to update ToC in this README, requires git-markdown-toc available inPATH
).
Please note that the make shell
rule requires cpupower
utility installed
that is invoked with sudo
to set cpufreq's performance governon
(it runs the CPU at the maximum frequence to eliminate throttling issues).
Makefiles contain recipes for building and executing tests with the
proper dependencies. Please use make run
(and make run2
where applicable).
The measurements are taken using analyze.rb
script:
$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]
Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:
$ ATTEMPTS=1 ../analyze.rb make run
Please use Dockerfile as a reference regarding which packages and tools are required.
For all (optional):
- Powercap for reading energy
counters in Linux (Debian package
powercap-utils
).
For Python:
- NumPy for matmul tests
(Debian package
python3-numpy
). - UltraJSON for JSON tests
(Debian package
python3-ujson
).
For C++:
- Boost for JSON tests
(Debian package
libboost-dev
). - JSON-C for JSON tests
(Debian package
libjson-c-dev
).
For Rust:
- libjq for jq test
(Debian packages
libjq-dev
,libonig-dev
and environment variableJQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/
).
For Java, Scala:
- Coursier for downloading Maven artifacts.
For Haskell:
- network for TCP connectivity between the tests and the test runner.
- raw-strings-qq for raw string literals used in tests.
For Perl:
- cpanminus for installing
modules from CPAN (Debian package
cpanminus
).
For Vala:
- JSON-GLib for JSON tests
(Debian package
libjson-glib-dev
).
Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:
- The test runner listens on localhost:9001;
- All messages are sent using TCP sockets closed immediately after the message has been sent;
- There are two messages sent from a test (it establishes the measurement
boundary):
- The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
- The end message with any content (mostly it's "stop" for consistency).
- The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.
If the test is compiled into a single binary, then two sections of
the Makefile
require changes:
- append a new target (the final binary location) into
executables
variable; - append the proper target rule.
If the test is compiled, but can't be executed directly as a binary, then
three sections of the Makefile
require changes:
- append a new target (the final artifact location) into
artifacts
variable; - append the proper target rule to compile the test;
- append
run[<target_artifact>]
rule to run the test.
If the test doesn't require compilation, then two sections of the Makefile
requires changes:
- append
run[<script_file>]
intoall_runners
variable; - append
run[<script_file>]
rule to run the test.