incubaid / crakoon Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 4.0 611 KB

A C client for Arakoon

License: Other

Shell 0.43% C 83.13% C++ 15.19% Python 1.25%

crakoon's People

Contributors

Stargazers

Watchers

Forkers

bas-at-amplidata redlicha openvstorage

crakoon's Issues

Multiget performance problem

We're seeing an issue with the crakoon multiget performance (against a single node arakoon cluster, using the arakoon 1.6.0 deb from arakoon.org).
the test case (code attached - it's C++ but uses the plain crakoon API) does 4096 multigets with a batch size of 1, value size is 4096:

./ara_multi_get --cluster test --nodes test_0,127.0.0.1,12345
keys: 4096, value_size: 4096, batch_size: 1
set took 4.94317 seconds -> 3.23679 MiB/s / 828.618 IOPS
get took 0.243652 seconds -> 65.6674 MiB/s / 16810.9 IOPS
multiget took 163.901 seconds -> 0.0976198 MiB/s / 24.9907 IOPS

This is a factor of 100 slower than the python client (from the arakoon git repo, branch 1.6):

In [3]: import ara_multi_get

In [4]: client = ara_multi_get.make_client()

In [5]: ara_multi_get.test_multigets??
Type: function
Base Class: <type 'function'>
String Form:<function test_multigets at 0x26406e0>
Namespace: Interactive
File: /home/arne/Projects/scrapyard/ara_multi_get.py
Definition: ara_multi_get.test_multigets(client, items, batchsize)
Source:
def test_multigets(client, items, batchsize):
keys = [ struct.pack('Q', k) for k in xrange(items) ]

logging.info("starting multigets")

with Timer() as t:
    for i in xrange(0, items / batchsize):
        j = i * batchsize
        client.multiGet(keys[j : j + batchsize])

logging.info("multigets with batchsize %d took %.03f sec: %.02f IOPS" %
             (batchsize,
              t.interval,
              items / t.interval))

In [6]: ara_multi_get.test_multigets(client, 4096, 1)
2013-09-20 10:27:40,816 starting multigets
2013-09-20 10:27:42,524 multigets with batchsize 1 took 1.707 sec: 2399.18 IOPS

This can also be observed with bigger batches (batchsize 64 -> C: ~ 1000 IOPS, Python ~3500 IOPS).

Arakoon client operation procedures don't have Doxygen docs

Some parts of the public API/header file have comments which are not tagged as Doxygen comments, and as such not included into the 'make doxygen' output. This should be fixed.

C++ bindings build fails using Clang 3.1

Build of the arakoonmm C++ module fails when using Clang 3.1, but succeeds using GCC 4.7.2:

$ make
...
/bin/sh ../libtool  --tag=CXX   --mode=compile clang++ -DHAVE_CONFIG_H -I. -I..   -DARAKOON_ASSERT=1 -DCRAKOON_MAJOR_VERSION=1 -DCRAKOON_MINOR_VERSION=1 -DCRAKOON_MICRO_VERSION=1 -DCRAKOON_VERSION=1.1.1 -DCRAKOON_GIT_VERSION=heads/1.2-0-gdb71dd4  -std=c++0x -g -O2 -Wall -Wextra -Werror -Wswitch-enum -g -O2 -MT libarakoonmm_1_0_la-arakoonmm.lo -MD -MP -MF .deps/libarakoonmm_1_0_la-arakoonmm.Tpo -c -o libarakoonmm_1_0_la-arakoonmm.lo `test -f 'arakoonmm.cpp' || echo './'`arakoonmm.cpp
libtool: compile:  clang++ -DHAVE_CONFIG_H -I. -I.. -DARAKOON_ASSERT=1 -DCRAKOON_MAJOR_VERSION=1 -DCRAKOON_MINOR_VERSION=1 -DCRAKOON_MICRO_VERSION=1 -DCRAKOON_VERSION=1.1.1 -DCRAKOON_GIT_VERSION=heads/1.2-0-gdb71dd4 -std=c++0x -g -O2 -Wall -Wextra -Werror -Wswitch-enum -g -O2 -MT libarakoonmm_1_0_la-arakoonmm.lo -MD -MP -MF .deps/libarakoonmm_1_0_la-arakoonmm.Tpo -c arakoonmm.cpp  -fPIC -DPIC -o .libs/libarakoonmm_1_0_la-arakoonmm.o
In file included from arakoonmm.cpp:29:
./arakoonmm.hpp:269:35: error: type 'const buffer_ptr' (aka 'const
      shared_ptr<arakoon::buffer>') does not provide a call operator
    buffer_ptr const buffer_ptr = buffer_ptr());
                                  ^~~~~~~~~~
arakoonmm.cpp:138:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_memory_set_hooks(hooks));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:241:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_value_list_iter_reset(iter_));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:250:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_value_list_iter_next(iter_, &value_size, &value_data));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:302:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_value_list_add(list_, value.size(), value.data()));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:371:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_key_value_list_iter_reset(iter_));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:382:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_key_value_list_iter_next(iter_, &key_size,...
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:487:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_sequence_add_set(sequence_, key.size(),...
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:494:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_sequence_add_delete(sequence_, key.size(), key.data()));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:502:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_sequence_add_assert(sequence_, key.size(),...
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:538:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_client_call_options_set_allow_dirty(options_,...
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:551:5: error: no matching function for call to 'rc_to_error'
    rc_to_error(arakoon_client_call_options_set_timeout(options_, timeout));
    ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:589:9: error: no matching function for call to 'rc_to_error'
        rc_to_error(arakoon_cluster_node_add_address_tcp(node_,...
        ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:593:9: error: no matching function for call to 'rc_to_error'
        rc_to_error(ARAKOON_RC_CLIENT_UNKNOWN_NODE);
        ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:602:9: error: no matching function for call to 'rc_to_error'
        rc_to_error(ARAKOON_RC_CLIENT_UNKNOWN_NODE);
        ^~~~~~~~~~~
arakoonmm.cpp:68:1: note: candidate function not viable: requires 2 arguments,
      but 1 was provided
rc_to_error(
^
arakoonmm.cpp:669:37: error: too few arguments to function call, expected 2,
      have 1
            arakoon::rc_to_error(rc1);
            ~~~~~~~~~~~~~~~~~~~~    ^
arakoonmm.cpp:67:1: note: 'rc_to_error' declared here
void
^
16 errors generated.
make[2]: *** [libarakoonmm_1_0_la-arakoonmm.lo] Error 1

Version info:

$ clang++ --version
clang version 3.1 (branches/release_31)
Target: x86_64-redhat-linux-gnu
Thread model: posix
$ g++ --version
g++ (GCC) 4.7.2 20121109 (Red Hat 4.7.2-8)
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Fix clang-analyzer detected issues

The clang-analyzer static code analyzer (version 3.1) detects 3 error paths in the crakoon code (at 71da712).Whilst they look mostly harmless or unreachable, it would be advisable to fix these.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.