The seq_stats from lynxzp

seq_stats's Introduction

This library designed for fast and efficient calculate basic statistical data of a big sequence of numbers, without loading all data into memory.

It directly calculate:

min
max
sum
mean
variance
standard deviation

And use t-digest algorithm to approximately calculate quantiles and build a chart.

T-digest is optional. Without t-digest, library computing statistic few times faster. As example six seconds for a one billion numbers.

$go test -bench . -benchmem                                                                                                                       goos: linux
goarch: amd64
pkg: github.com/lynxzp/seq_stats
cpu: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
BenchmarkStat/input_size_1000-8                   276945              3612 ns/op               0 B/op          0 allocs/op
BenchmarkStat/input_size_1000000-8                   328           3700582 ns/op               0 B/op          0 allocs/op
BenchmarkStat/input_size_1000000000-8                  1        3643973531 ns/op               0 B/op          0 allocs/op
PASS
ok      github.com/lynxzp/seq_stats 6.370s

Example

import "github.com/lynxzp/seq_stats"

const tDigestEnabled = true
s := seq_stats.New[int64](tDigestEnabled)
for i := 0; i < 1000000; i++ {
    s.Add(int64(i))
}

fmt.Println(s.Min())
fmt.Println(s.Quantile(0.5))
fmt.Println(s)

writer := bytes.NewBuffer([]byte{})
const pointCount = 1000
_ = s.DrawPNG(writer, pointCount)

result:

0
499999.5
min: 0, max: 999999, avg: 499999.5, rms: 577349.8, stddev: 288675 sum: 4.999995e+11 amount: 1000000
0.1%: 999.5, 1%: 9999.5, 10%: 99999.5, 50%: 499999.5, 90%: 899999.5, 99%: 989999.5, 99.9%: 998999.5

Recommend Projects