dataartisans / flink-benchmarks Goto Github PK

Our latest flink benchmark Jenkins builds are failing with below error:

[ERROR] Failed to execute goal on project flink-hackathon-benchmarks: Could not resolve dependencies for project org.apache.flink.benchmark:flink-hackathon-benchmarks:jar:0.1: Failure to find org.apache.flink:flink-tests_2.11:jar:tests:1.10-SNAPSHOT in https://repository.apache.org/content/repositories/snapshots/

Which is due to the branching of 1.9 has updated our flink project version to 1.10 while still no 1.10 snapshot deployed yet. I tried to help deploy but it turned out with no access. We need someone with the privilege help deploy the snapshot for the new version.

What's more, to prevent such issue happen again, we need to document it somewhere that a deploy of snapshot is necessary when branching new releases and update the snapshot version.

Currently, the statebanchmark did not delete the temp directory after the benchmark teardown, so we'll have many empty directory in /tmp,

https://github.com/dataArtisans/flink-benchmarks/blob/master/src/main/java/org/apache/flink/state/benchmark/ListStateBenchmark.java#L119

This is introduced in PR#13.

Shouldn't this be listState.add ?

Please correct me if I'm wrong. Thanks.

cc @carp84

I wonder if these benchmarks only work with local mode, or we could modify and apply them on yarn cluster or other deploy environment.

Results of ListStateBenchmark are not stable:

# Benchmark: org.apache.flink.state.benchmark.ListStateBenchmark.listAddAll
# Parameters: (backendType = HEAP)

# Run progress: 5.56% complete, ETA 00:35:16
# Fork: 1 of 3
# Warmup Iteration   1: 763.432 ops/ms
# Warmup Iteration   2: 114.673 ops/ms
# Warmup Iteration   3: 169.797 ops/ms
# Warmup Iteration   4: 805.482 ops/ms
# Warmup Iteration   5: 346.139 ops/ms
# Warmup Iteration   6: 1292.193 ops/ms
# Warmup Iteration   7: 23.363 ops/ms
# Warmup Iteration   8: 756.512 ops/ms
# Warmup Iteration   9: 845.994 ops/ms
# Warmup Iteration  10: 86.222 ops/ms
Iteration   1: 725.139 ops/ms
Iteration   2: 1397.114 ops/ms
Iteration   3: 311.249 ops/ms
Iteration   4: 312.853 ops/ms
Iteration   5: 6.339 ops/ms
Iteration   6: 1179.413 ops/ms
Iteration   7: 1380.267 ops/ms
Iteration   8: 1391.004 ops/ms
Iteration   9: 571.861 ops/ms
Iteration  10: 31.509 ops/ms

The same applies for RocksDB, which is visible in averages:
http://codespeed.dak8s.net:8000/timeline/#/?exe=3&ben=listAddAll.ROCKSDB&env=2&revs=1000&equid=off&quarts=on&extr=on

Does this benchmark have a GC issue?

CC @carp84

Hi all

From my understanding, the benchmark for state-backend should only compare the performance of combined state operations, but the operation to benchmark within JMH contains the time of submitting job, deploying tasks and so on.
Why not just compare the standalone state-backend performance without influence from other Flink components?