Code Monkey home page Code Monkey logo

eredis's Introduction

Build Status

EREDIS

Eredis is a C client library built over Hiredis. It is lightweight, high performance, reentrant and thread-safe.
It aims to provide features for real production environment and keep it simple.

For write commands (SET, HSET, EXPIRE, ...):

  • Async commands via asynchronous events (libev)
  • Async commands mirroring across multiple redis servers (shared-nothing)

For any commands:

  • Thread-safe high speed pool of blocking connections: efficient persistent connections
  • Automatic and seamless fail-over and reconnect

Integration:

  • Embedded Hiredis ensures the full Redis protocol support
  • Reply structure is the Hiredis efficient redisReply
  • Eredis allows you to switch from a single host mode to a an efficient shared-nothing/fail-over mode by just activating a new host.

The integrated master-slave mechanism in Redis works ok for a small number of nodes or workload, but it could kill a master node bandwidth in high load. The many-to-many mechanism permits a more efficient and reliable workload by obviously needing exactly the same bandwidth on each node.

Compile

git clone <eredis_git_url>
cd eredis
git submodule update --init
cmake .
make
# optional
make doc
make install

Test

If you have Redis installed, and are not cross-compiling, you can run the test suite that comes with Eredis after you have built the library itself (using the steps in the compile section, above) with the test target.

make test

Usage

Each function is described in the C code (doxygen).
The raw list is available in the unique header file: eredis.h.

init

#include <eredis.h>

eredis_t *e = eredis_new();

/* Set timeout - default 5000ms */
eredis_timeout( e, 200 );

/* Set max readers - default 10 */
eredis_r_max( e, 50 );

/* Set retry for reader - default 1 */
eredis_r_retry( e, 1 );

add redis targets

The first host provided becomes the "prefered" one.
The readers will always reconnect to this one in case of a down/up event.

/* manually */
eredis_host_add( e, "/var/run/eredis.sock", 0 );
eredis_host_add( e, "host2", 6379 );
eredis_host_add( e, "host3", 6379 );

/* via configuration file - one line per host:port */
eredis_host_file( e, "my-hosts.conf" );

add post-connection requests (beta)

For 'AUTH' or any one-time command needed to be executed after connect. Eredis ensures that these commands are executed in order before any other reader or writer commands.

char *pwd = "mysecret";
eredis_pc_cmd( e, "AUTH mysecret" );
eredis_pc_cmd( e, "AUTH %s", pwd );
eredis_pc_cmd( e, "SCRIPT DEBUG YES" );

launch the async loop

Mandatory for using async writes or auto-reconnection to the "prefered" host.

/* in its own thread - blocking call */
eredis_run( e );

/* in a eredis managed thread - non-blocking call */
eredis_run_thr( e );

sync requests (to get reply)

eredis_reader_t *r;
eredis_reply_t *reply; /* aka redisReply from hiredis */

/* get a reader */
r = eredis_r( e );

/* Add one request (pipelining) */
eredis_r_append_cmd( reader, "GET key1");

/* Add one request and process,
   return key1 reply (the first one from queue) */
reply = eredis_r_cmd( reader, "GET key2");

if ( I_NEED_ALL_REPLIES ) {
 /* current 'reply' hosts 'key1' reply */

 /* Get key2 reply */
 reply = eredis_r_reply( reader );
 ...
}

if ( I_NEED_TO_SEND_MORE ) {
 /* this one fetches all possible replies left in the pipelining */
 eredis_r_clear( reader );

 /* reader is ready to perform a new batch of r_cmd+replies */
 ..
}

/* Release the reader */
eredis_r_release( r );

subscribe requests (beta, blocking)

eredis_reader_t *r;
eredis_reply_t *reply; /* aka redisReply from hiredis */

/* get a reader */
r = eredis_r( e );

/* Add one request */
eredis_r_append_cmd( reader, "SUBSCRIBE chan1");

/* Get subscribe requests.
   Replies from SUBSCRIBE commands are omitted.
   Eredis manages reconnect and re-subscribe to channels */

while (( reply = eredis_r_subscribe( reader ) ))
{
 ...
 if ( HAVE_ENOUGH )
  break;
}

/* Add a new channel */
eredis_r_append_cmd( reader, "SUBSCRIBE chan2");

while (( reply = eredis_r_subscribe( reader ) ))
{
 ...
 if ( HAVE_ENOUGH )
  break;
}

/* 'release' is not disconnecting, just unsubscribe properly */
eredis_r_cmd( reader, "UNSUBSCRIBE" );

/* Release the reader */
eredis_r_release( r );

async requests (no reply, non-blocking)

eredis_w_cmd( e, "SET key1 10" );

stop

/* Exit the event loop (from any thread) */
eredis_shutdown( e );

/* Stop the event loop and threadif needed,
   close connections and release memory */
eredis_free( e );

Day-to-day

When a Redis server goes down, Eredis async loop will detect it and retry to reconnect every second HOST_DISCONNECTED_RETRIES (10) times (= 10 seconds).
After this, it will retry once every HOST_FAILED_RETRY_AFTER (20) seconds.

To avoid data loss, if all specified Redis server are down, Eredis will keep in memory the last unsent QUEUE_MAX_UNSHIFT (10000) commands.

If a Redis server goes down and up, it could be needed to resynchronize with an active node. The master-slave mechanism is perfect for that.
In redis.conf, add 'slave-read-only no'.
After the Redis server start, make a "redis-cli SLAVEOF hostX".
Once 'redis-cli info Replication' pops a 'master_sync_in_progress:0', it's done.
make a "redis-cli SLAVEOF no one" and it comes back to a 'master' status.
This process can easily be scripted.

AUTHOR

Eredis is used on nearly 500 servers at Eulerian Technologies.
It is written and maintained by Guillaume Fougnies (guillaume at eulerian dot com).
Redis and Hiredis are great!
We hope you will find Eredis great.
It is released under the BSD license.

eredis's People

Contributors

guillaumef avatar xorangekiller avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

eredis's Issues

about payload size

Hello,
when i do a eredis_r_cmd , with a redis get , that 's sending a big payload , ~ 8Mb , my progam segfault at eredis_r_cmd ( i cannot test the eredis_r_cmd reply ) , do you have some static buffer ? or hiredis ?

Regards,
Nicolas

End of replies from eredis_r_reply(r)

I am doing a variable number of eredis_r_append_cmd() and a final reply=eredis_r_cmd().

When checking replies with reply = eredis_r_reply( r ), I have no idea how many commands were issued (I could count, but eredis through r->cmds_replied already knows this). Fortunately reply==NULL when no more replies are available, so I can just iterate through the replies. Unfortunately it prints out an error message. Is getting to the end of replies really an error? Is there something else I'm missing?

I'd just do the same check in my code for (r->cmds_replied >= r->cmds_nb), but apparently that is not available outside the API.

From rw.c:

if (r->cmds_replied >= r->cmds_nb) {
    fprintf(stderr,
            "eredis: api misuse: all cmds are already replied: %d/%d\n",
            r->cmds_replied, r->cmds_nb);
    return NULL;
}

Pub/Sub Async ?

Is it possible to use it for Pub/Sub Async?

haven't found a way to do it, or should I use directly Hiredis with redisAsyncContext?

tried some examples but no luck

src/eredis.c:202: error: redefinition of typedef ‘eredis_reader_t’

Hello!

[root@gerasim eredis]# make
[ 20%] Built target hiredis-inc
[ 40%] Built target hiredis-src
[ 50%] Building C object src/CMakeFiles/eredis.dir/eredis.c.o
/tmp/c/redis/eredis/src/eredis.c:202: error: redefinition of typedef ‘eredis_reader_t’
/tmp/c/redis/eredis/include/eredis.h:59: note: previous declaration of ‘eredis_reader_t’ was here
/tmp/c/redis/eredis/src/eredis.c:238: error: redefinition of typedef ‘eredis_t’
/tmp/c/redis/eredis/include/eredis.h:58: note: previous declaration of ‘eredis_t’ was here
make[2]: *** [src/CMakeFiles/eredis.dir/eredis.c.o] Error 1
make[1]: *** [src/CMakeFiles/eredis.dir/all] Error 2
make: *** [all] Error 2

Please tell how it can be compiled? Or i'm doing something wrong?

Stable version releases?

I can't seem to find anywhere to get stable packaged releases for eredis anywhere (either in Debian's repositories or here in the GitHub releases tab). Am I merely blind or are users supposed to simply use master from source? :)

eredis connects but does not SET anything via async connection

e = eredis_new();
eredis_timeout( e, 200 );
eredis_r_max( e, 50 );
eredis_r_retry( e, 1 );

eredis_host_add( e, "127.0.0.1", 6379 );

eredis_pc_cmd( e, "AUTH passwd" );
eredis_pc_cmd( e, "SELECT 1");
eredis_run_thr( e );
eredis_w_cmd( e, "SET testkey 10" );

With this snippet (I trimmed variable declarations, by the way) I am able to connect redis-server as I checked via connection tools and also by redis itself. However, testkey does not SET. The most interesting thing is, sometimes (randomly) I see that this teskey declared successfully without reason. Like redis get the key but do some delay. However, this is not simply a delay. It happens randomly. After few minutes maybe that key will be set.

Is there any reason to have such behavior on my end? Redis server is v4.

eredis ipv6

hello,
it seems eredis_host_file( e, "my-hosts.conf" ) does not work for ipv6 , parsing : seems bad ,
can you correct ?

Regards,
Nicolas Prochazka

Does it can build on Centos

OS:CentOS release 6.9
when I type

cmake .

output:

-- Could NOT find libev (missing: LIBEV_INCLUDE_DIR LIBEV_LIBRARY)
CMake Error at src/CMakeLists.txt:90 (MESSAGE):
Could not find required package libev

I install "libev-devel.x86_64" , then "cmake ." again
output:

-- Could NOT find libev (missing: LIBEV_INCLUDE_DIR)
CMake Error at src/CMakeLists.txt:90 (MESSAGE):
Could not find required package libev

I check the "CMakeError.log", find

/usr/bin/ld: cannot find -lpthreads
collect2: ld returned 1 exit status

which step I do was wrong, and what should I do to resolve the problem.
Thanks!

publish/subscribe

What is the right way to implement publish/subscribe on top of eredis?

I tried sending a 'subscribe' command, and it works fine, but I can't call eredis_r_reply() in a blocking manner to wait for a response that hasn't come yet.

Example

Can you put some examples ?
Is eredis_reply_free must be use with simple eredis_r_cmd ?
How count reader ( thread ?) that opens in pool connection ?

Regards,
Nicolas Prochazka

error in documentation

hello,
we can read in ReadMe :
/* release a reader */
r = eredis_r_release( e );

e or reader ?

Regards,
Nicolas

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.