ryanphuang / imembench Goto Github PK

Generic benchmarks for various in-memory storage systems

Shell 1.23% C++ 88.62% Java 4.65% Python 4.14% Makefile 0.10% C 0.77% Batchfile 0.04% Elixir 0.01% HTML 0.44%

imembench's Introduction

imembench

Various in-memory storage systems exist today that offer similar functionality but with different features (some more performant, others more reliable). Each of the systems usually has its own APIs, making the benchmark results of them often incomparable. The goal of imembench is to extract and create a common wrapper of the interfaces exposed by these systems, and then drive these systems with almost-the-same workloads by exercising this common interface.

Implementation-wise, we need to use a common language that can talk to these systems in the form of either the source language or bindings.

checkout repository:

after clone the main repository, run git submodule update --init --recursive to pull the git submodules for building.

build instructions:

Build dependencies:
- RAMCloud: at project root, cd bindings/ramcloud/c++ && make
  - if the generated objects are in bindings/ramcloud/c++/obj, rename the obj directory to obj.master
- Redis: at project root, cd bindings/redis/c++/hiredis && make
- Tachyon: at project root, cd bindings/tachyon/c++ && make
Build imembench:
- cd loadgenerator && make

use benchmark

First, source env.sh

Then, copy the samplebench.ini (in loadgenerator directory) to imembench.ini and modify the configurations accordingly.

Usage: ./imembench [OPTIONS] [tachyon|ramcloud|redis|all]

  OPTION
        -l, --list                   list available benchmarks and supported targets
        -c, --config_file            configuration file (default imembench.ini) for the benchmark
        -w, --workload_file          workload trace file to use instead of the default benchmark
        -b, --benchmark              comma-separated list of benchmarks to run on the target systems
        -t, --target                 comma-separated list of target system to evaluate

  EXAMPLE
        ./imembench -c imembench.ini redis   # care, run all benchmarks, very slow
        ./imembench -c imembench.ini --benchmark uniformRandomRW redis
        ./imembench -c imembench.ini --workload_file trace.ycsb --benchmark ycsbReplay --target redis,ramcloud

supported targets and benchmarks

Supported targets:

ramcloud
tachyon
redis

Supported benchmarks:

ycsbReplay
uniformRandomRW
zipfRandomRW
uniformRandomReadMost
zipfRandomReadMost
uniformRandomWriteMost
zipfRandomWriteMost
uniformRandomRWSmallObj
zipfRandomRWSmallObj
uniformRandomReadMostSmallObj
zipfRandomReadMostSmallObj
uniformRandomWriteMostSmallObj
zipfRandomWriteMostSmallObj
uniformRandomRWLargeObj
zipfRandomRWLargeObj
uniformRandomReadMostLargeObj
zipfRandomReadMostLargeObj
uniformRandomWriteMostLargeObj
zipfRandomWriteMostLargeObj

existing interfaces:

RAMCloud:
- createTable(name)
- dropTable(name)
- getTableId(name)
- read(tableId, key, keyLength, value, rejectRules, ...)
- write(tableId, key, keyLength, buf, length, ...)
- increment(tableId, key, keyLength, incrementValue, ...)
- remove(tableId, key, keyLength, ...)
Tachyon (http://tachyon-project.org/api/java/index.html):
- int read()
- int read(byte[] b)
- int read(byte[] b, int off, int len)
- void seek(long pos)
- skip(long n)
- write(byte[] b)
- write(byte[] b, int off, int len)
- write(int b)
- flush()
- mkdir(TachyonURI path)
- rename(TachyonURI srcPath, TachyonURI dstPath)
Redis:
- http://redis.io/commands

existing languages:

RAMCloud:
- Source: C/C++
- Bindings: Python, Java
Tachyon:
- Source: Java
- Bindings: TBA
Redis:
- Source: C/C++
- Bindings: C, C#, C++, Clojure, Common List, D, Dart, Elixir, emacs lisp, Erlang, Fancy, GNU Prolog, Go, Haskell, haXe, Io, Java, Lua, Matlab, Nimrod, Node.js, Objective-C, OCaml, Perl, PHP, Pure Data, Python, Rebol, Ruby, Rust, Scala, Scheme, Smalltalk, Tcl, VCL. (http://redis.io/clients)

imembench's People

Contributors

Stargazers

Watchers

Forkers

abulila

imembench's Issues

Runtime failure of the master node after instrumentation

Encounter runtime failures after running the instrumented master node (using java-allocation-instrumenter-3.0).

Exception in thread "main" java.lang.IllegalArgumentException: Strange Collection type java.util.ArrayList: can not determine type parameters
    at com.fasterxml.jackson.databind.type.TypeFactory._collectionType(TypeFactory.java:904)
    at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:722)
    at com.fasterxml.jackson.databind.type.TypeFactory._constructType(TypeFactory.java:371)
    at com.fasterxml.jackson.databind.type.TypeFactory.constructType(TypeFactory.java:342)
    at com.fasterxml.jackson.databind.cfg.MapperConfig.constructType(MapperConfig.java:241)
    at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:442)
    at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:599)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:93)
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2682)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2630)
    at tachyon.master.JsonObject.withParameter(JsonObject.java:110)
    at tachyon.master.ImageElement.withParameter(ImageElement.java:53)
    at tachyon.master.InodeFolder.writeImage(InodeFolder.java:256)
    at tachyon.master.MasterInfo.writeImage(MasterInfo.java:2417)
    at tachyon.master.Image.create(Image.java:59)
    at tachyon.master.Journal.createImage(Journal.java:78)
    at tachyon.master.MasterInfo.init(MasterInfo.java:1780)
    at tachyon.master.TachyonMaster.setup(TachyonMaster.java:216)
    at tachyon.master.TachyonMaster.start(TachyonMaster.java:277)
    at tachyon.master.TachyonMaster.main(TachyonMaster.java:73)

Discussion on allocation of Tachyon

I wanna invoke a discussion on how to measure the allocation overhead of Tachyon.

Assumptions

Foremost, according to my understanding, only the allocation overhead of workers (who stores real data) matters; the master only maintains metadata so it's allocation overhead is not interesting. Correct me if I'm wrong @stormspirit, @jakemask

Attempts

I managed to apply the java-allocation-instrumenter (https://github.com/google/allocation-instrumenter) on Tachyon workers, and passed all the tests.

Difficulties

However, I encountered the difficulties to analyze the recorded overhead. First, the instrumenter does not tell the time overhead but only records the size of each allocated objects.

Second, the method used by instrumenter is to insert instructions after each new statement, which does not tell the real allocation overhead. This is a general problem of instrumentation based on new or malloc

Long story short, there are two big problems (I don't know how to solve),

1. The recorded log does not tell the real size of the object

Consider the following code snippets,

    new String("foo");
    new String("fooo");
    new String("foooo");
    new String("fooooo");
    new String("foooooo");
    new String("fooooooo");

And the logs show,

Allocated object foo of type java/lang/String with size 24
Allocated object fooo of type java/lang/String with size 24
Allocated object foooo of type java/lang/String with size 24
Allocated object fooooo of type java/lang/String with size 24
Allocated object foooooo of type java/lang/String with size 24
Allocated object fooooooo of type java/lang/String with size 24
Allocated object java.lang.Object@76724124 of type java/lang/Object with size 16
Allocated object [C@78b5a19a of type char with size 48 (an array of size 13)
Allocated object com.google.monitoring.runtime.instrumentation.asm.ClassReader@3988aba1 of type com/google/monitoring/runtime/instrumentation/asm/ClassReader with size 32
Allocated object [C@5d97b6d5 of type char with size 80 (an array of size 32)
Allocated object [C@8fb758c of type char with size 32 (an array of size 6)
Allocated object [C@250e364d of type char with size 80 (an array of size 30)
Allocated object [C@59a754ee of type char with size 32 (an array of size 6)
Allocated object [C@39667011 of type char with size 72 (an array of size 27)
Allocated object [C@6a6235a4 of type char with size 64 (an array of size 21)
Allocated object [C@3e0adafb of type char with size 24 (an array of size 3)
Allocated object [C@5de9e9f0 of type char with size 96 (an array of size 37)
Allocated object [C@5705fca4 of type char with size 136 (an array of size 59)
Allocated object [C@2cb12cf5 of type char with size 72 (an array of size 25)
Allocated object [C@4484f69a of type char with size 24 (an array of size 4)
Allocated object [C@56302193 of type char with size 24 (an array of size 1)
Allocated object [C@2ea8072d of type char with size 40 (an array of size 11)
Allocated object [C@bdb4089 of type char with size 64 (an array of size 21)
Allocated object [C@6808574e of type char with size 32 (an array of size 6)
Allocated object [C@3f63cf65 of type char with size 96 (an array of size 38)
Allocated object [C@511c26cb of type char with size 56 (an array of size 17)
......

From the first 4 lines, we can see the String object actually have the same size. What differs is the field objects inside the strings. However, as the instrumenter does not provide any more informative interface, this is pretty much the only thing we can get. I don't know how to leverage such information, except do a type-based analysis.

2. The instrumenter only looks at the `new` statement.

In fact, if Tachyon or any other system performs some pre-allocation (e.g., allocating a pool of memory dur ing initialization), and then return the memory blocks when the client requests files/blocks. The instrumenter would tell us nothing because there is no new statements invoked at that time.

Thoughts/Solutions

Luckly, in Tachyon, memory allocation related code is well maintained in the directory
core/src/main/java/tachyon/worker/allocation

They implemented three allocation strategy: RR, MaxFree, and Random, all of them inherits the abstract class AllocateStrategyBase.

Basically, the only method requires to implement is getStorageDir() which

 33   /**
 34    * Allocate space on StorageDirs. It returns the affordable StorageDir according to
 35    * AllocateStrategy, null if no enough space available on StorageDirs.
 36    *
 37    * @param storageDirs candidates of StorageDirs that space will be allocated in
 38    * @param userId id of user
 39    * @param requestSizeBytes size to request in bytes
 40    * @return StorageDir assigned, null if no StorageDir is assigned
 41    */

Though the algorithm of selecting the affordable StorageDir are different, the key operation is ``requestSpace()` which get the space from the affordable storage.

So my idea is to add time counters in getStorageDir() so that we can know the allocation overhead.

kvtracer issue

When I try to run the following command in the kvtracer directory
mvn -pl kvtracer -am clean package

I get the output shown here: http://pastebin.com/DuvV5vLe

My directory structure is:
ycsb-0.7.0/kvtracer