Code Monkey home page Code Monkey logo

redisgears's Introduction

Triggers and Functions

GitHub Builds CircleCI Documentation

A Redis module that allows running a JS functions inside the Redis processes. The JS code is execute use V8 JS engine.

Notice, Triggers and Functions is still under active development and not yet GA. The API might (and probably will) change at the final GA version.

Run Using Docker

docker run -p 6379:6379 redislabs/redisgears:edge

How do I Redis?

Learn for free at Redis University

Build faster with the Redis Launchpad

Try the Redis Cloud

Dive in developer tutorials

Join the Redis community

Work at Redis

Build

With Docker

Run the following on the main directory:

> docker build -t redisgears2 .

Then run the built image:

> docker run -p 6379:6379 redisgears2

From Source

See the build page for more information.

Getting started

Run JS code

The API exposed by the module is very similar to the way Redis Functions work. Let's write a simple hello world function that returns the string hello world:

#!js name=lib api_version=1.0

redis.registerFunction('hello_world', function(){
    return 'hello_world';
});

The first line indicates the engine to use (js) and the library name (lib). The rest is the library code.

Assuming we put the following code in the file lib.js, we can register our function in Redis using the TFUNCTION LOAD command:

> redis-cli -x TFUNCTION LOAD < ./lib.js
OK

And now we can execute our function using TFCALL command, the command gets the library name and the function name . separated:

> redis-cli TFCALL lib.hello_world 0
"hello_world"

Notice that the TFCALL command arguments are very close to the Redis FCALL command arguments. The only difference is that, for Triggers and Functions, the command also requires the library name. The 0 represents the number of keys that will follow, which in our case is 0.

Calling Redis Commands Inside Triggers and Functions

It is possible to call Redis commands inside Triggers and Functions. The function gets as the first argument a client object that allows interaction with Redis using the call function. The following example executes a simple PING command and returns the result:

#!js name=lib api_version=1.0

redis.registerFunction('my_ping', function(client){
    return client.call('ping');
});

If we will try to send it to our running Redis instance, we will get the following error:

> redis-cli -x TFUNCTION LOAD < ./lib.js
(error) Library lib already exists

We get the error because the library with the same name already exists, we can use the REPLACE argument to replace the library with the new code:

> redis-cli -x TFUNCTION LOAD UPGRADE < ./lib.js
OK

And now we can invoke my_ping using TFCALL :

> redis-cli TFCALL lib.my_ping 0
"PONG"

What's next?

redisgears's People

Contributors

alonre24 avatar andresrinivasan avatar banker avatar chayim avatar dallastexas92 avatar dengliming avatar dependabot[bot] avatar dmaier-redislabs avatar dvirdukhan avatar dwdougherty avatar filipecosta90 avatar gkorland avatar iddm avatar itamarhaber avatar justincastilla avatar k-jo avatar kamran-redis avatar kerneljake avatar leibale avatar mehamasum avatar meirshpilraien avatar ofirmos avatar quintonparker avatar rafie avatar rrelledge avatar simonprickett avatar snyk-bot avatar thomascaudron avatar tomerhekmati avatar viktarstarastsenka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

redisgears's Issues

Add sugar syntax to pyexecute command

  • count - count the number of records
  • distinct count - count the distinct number of records
  • count by - group by + count reducer
  • distinct count by - group by + distinct count reducer
  • aggregate(zeroValue, seqOp, combOp) - like accumulate, but it gets the zero value, the accumulate, and combine accumulators (for better cluster support)
  • aggregateby - same as aggregate but includes groupby
  • Sort

`make get_deps` requires root

...
running build_scripts
ln -fs /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata.py /usr/lib/python2.7/
ln: /usr/lib/python2.7//_sysconfigdata.py: Operation not permitted

The null character and possibly others

We have a problem with binary data in Strings:

127.0.0.1:6379> SET foo "a\x00z"
OK
127.0.0.1:6379> GET foo
"a\x00z"
127.0.0.1:6379> RG.PYEXECUTE "gearsCtx().foreach(lambda x: redisgears.executeCommand('SET', 'bar', str(x['value']))).map(lambda x: str(x)).run('foo')"
1) 1) "TotalExecutionDuration"
   2) (integer) 98000
   3) "TotalReadDuration"
   4) (integer) 37000
2) 1) "{'value': 'a\\x00z', 'key': 'foo'}"
127.0.0.1:6379> GET bar
"a"

Document commands

PYEXECURTE
DUMPEXECUTIONS
DROPEXECUTION

... anything missing? should go in docs/commands.md

lay out --> redistimeseris.io is good example

Crash on Gears + Search

127.0.0.1:6379> MODULE LOAD /home/guy/redislabsmodules/RedisGears/redisgears.so
OK
(1.53s)
127.0.0.1:6379> MODULE LOAD /home/guy/redislabsmodules/RedisSearch/build/redisearch.so
OK
(0.93s)
127.0.0.1:6379> hset aaa aa bb cc dd
(integer) 2
127.0.0.1:6379> ft.create ads SCHEMA aa TEXT cc TEXT
OK
127.0.0.1:6379> FT.ADDHASH ads aaa 1.0
OK
127.0.0.1:6379> 
1081:M 11 Mar 2019 18:03:04.488 * <ft> Initialized thread pool!
1081:M 11 Mar 2019 18:03:04.488 * Module 'ft' loaded from /home/guy/redislabsmodules/RedisSearch/build/redisearch.so
free(): invalid pointer
Aborted (core dumped)
837:C 11 Mar 17:57:47.201 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
837:C 11 Mar 17:57:47.235 # Redis version=4.0.9, bits=64, commit=3150c672, modified=0, pid=837, just started
837:C 11 Mar 17:57:47.236 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
837:M 11 Mar 17:57:47.290 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
837:M 11 Mar 17:57:47.295 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
837:M 11 Mar 17:57:47.295 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.9 (3150c672/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 837
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

837:M 11 Mar 17:57:47.575 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
837:M 11 Mar 17:57:47.577 # Server initialized
837:M 11 Mar 17:57:47.578 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
837:M 11 Mar 17:57:47.578 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
837:M 11 Mar 17:57:47.580 * Ready to accept connections
/home/guy/redislabsmodules/RedisGears/redisgears.so
837:M 11 Mar 17:57:55.374 * <rg> RedisGears version 0.1.0, git_sha=a96fe769ecc6bb6066a7bcc682e68b357c7244d2
837:M 11 Mar 17:57:55.390 * <rg> PythonHomeDir:/usr/bin/
could not initialize RedisDL_TensorCreate
837:M 11 Mar 17:57:55.392 # <rg> could not initialize RediDL api, running without DL support.
could not initialize RediSearch_CheckApiVersionCompatibility
837:M 11 Mar 17:57:55.393 # <rg> could not initialize RediSearch api, running without Search support.
837:M 11 Mar 17:57:56.733 * Module 'rg' loaded from /home/guy/redislabsmodules/RedisGears/redisgears.so
837:M 11 Mar 17:58:01.676 * <ft> Low level api version 1 initialized successfully
837:M 11 Mar 17:58:01.676 * <ft> RediSearch version 1.4.2 (Git=v1.4.2-270-gb74f61c5)
837:M 11 Mar 17:58:01.679 * <ft> concurrency: ON, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, search pool size: 20, index pool size: 8, 
837:M 11 Mar 17:58:02.477 * <ft> Initialized thread pool!
837:M 11 Mar 17:58:02.499 * Module 'ft' loaded from /home/guy/redislabsmodules/RedisSearch/build/redisearch.so
==837== Invalid read of size 8
==837==    at 0x1E078C: je_arena_mapbitsp_read (arena.h:649)
==837==    by 0x1E078C: je_arena_mapbits_get (arena.h:656)
==837==    by 0x1E078C: je_arena_mapbits_binind_get (arena.h:713)
==837==    by 0x1E078C: je_arena_salloc (arena.h:1218)
==837==    by 0x1E078C: je_isalloc (jemalloc_internal.h:876)
==837==    by 0x1E078C: je_malloc_usable_size (jemalloc.c:2483)
==837==    by 0x143C1D: zfree (zmalloc.c:194)
==837==    by 0x96795FD: VVW_Cleanup (varint.h:62)
==837==    by 0x96795FD: ByteOffsetWriter_Cleanup (byte_offsets.h:55)
==837==    by 0x96795FD: AddDocumentCtx_Free (document.c:320)
==837==    by 0x96796AA: doReplyFinish (document.c:179)
==837==    by 0x96796AA: replyCallback (document.c:184)
==837==    by 0x1AA73F: moduleHandleBlockedClients (module.c:3550)
==837==    by 0x139DF6: beforeSleep (server.c:1222)
==837==    by 0x136A0D: aeMain (ae.c:497)
==837==    by 0x13354B: main (server.c:3894)
==837==  Address 0x50005a0 is 32 bytes before a block of size 48 in arena "client"
==837== 
==837== Invalid read of size 8
==837==    at 0x1E078C: je_arena_mapbitsp_read (arena.h:649)
==837==    by 0x1E078C: je_arena_mapbits_get (arena.h:656)
==837==    by 0x1E078C: je_arena_mapbits_binind_get (arena.h:713)
==837==    by 0x1E078C: je_arena_salloc (arena.h:1218)
==837==    by 0x1E078C: je_isalloc (jemalloc_internal.h:876)
==837==    by 0x1E078C: je_malloc_usable_size (jemalloc.c:2483)
==837==    by 0x143C25: zfree (zmalloc.c:194)
==837==    by 0x96795FD: VVW_Cleanup (varint.h:62)
==837==    by 0x96795FD: ByteOffsetWriter_Cleanup (byte_offsets.h:55)
==837==    by 0x96795FD: AddDocumentCtx_Free (document.c:320)
==837==    by 0x96796AA: doReplyFinish (document.c:179)
==837==    by 0x96796AA: replyCallback (document.c:184)
==837==    by 0x1AA73F: moduleHandleBlockedClients (module.c:3550)
==837==    by 0x139DF6: beforeSleep (server.c:1222)
==837==    by 0x136A0D: aeMain (ae.c:497)
==837==    by 0x13354B: main (server.c:3894)
==837==  Address 0x50005a0 is 32 bytes before a block of size 48 in arena "client"
==837== 
==837== Invalid read of size 8
==837==    at 0x1DC6AC: je_arena_mapbitsp_read (arena.h:649)
==837==    by 0x1DC6AC: je_arena_mapbits_get (arena.h:656)
==837==    by 0x1DC6AC: je_arena_mapbits_binind_get (arena.h:713)
==837==    by 0x1DC6AC: je_arena_salloc (arena.h:1218)
==837==    by 0x1DC6AC: je_isalloc (jemalloc_internal.h:876)
==837==    by 0x1DC6AC: ifree (jemalloc.c:1740)
==837==    by 0x1DC6AC: je_free (jemalloc.c:1839)
==837==    by 0x96795FD: VVW_Cleanup (varint.h:62)
==837==    by 0x96795FD: ByteOffsetWriter_Cleanup (byte_offsets.h:55)
==837==    by 0x96795FD: AddDocumentCtx_Free (document.c:320)
==837==    by 0x96796AA: doReplyFinish (document.c:179)
==837==    by 0x96796AA: replyCallback (document.c:184)
==837==    by 0x1AA73F: moduleHandleBlockedClients (module.c:3550)
==837==    by 0x139DF6: beforeSleep (server.c:1222)
==837==    by 0x136A0D: aeMain (ae.c:497)
==837==    by 0x13354B: main (server.c:3894)
==837==  Address 0x50005a0 is 32 bytes before a block of size 48 in arena "client"
==837== 
==837== Invalid read of size 8
==837==    at 0x1DC712: je_arena_mapbitsp_read (arena.h:649)
==837==    by 0x1DC712: je_arena_mapbits_get (arena.h:656)
==837==    by 0x1DC712: je_arena_dalloc (arena.h:1264)
==837==    by 0x1DC712: je_idalloctm (jemalloc_internal.h:1005)
==837==    by 0x1DC712: je_iqalloc (jemalloc_internal.h:1029)
==837==    by 0x1DC712: ifree (jemalloc.c:1745)
==837==    by 0x1DC712: je_free (jemalloc.c:1839)
==837==    by 0x96795FD: VVW_Cleanup (varint.h:62)
==837==    by 0x96795FD: ByteOffsetWriter_Cleanup (byte_offsets.h:55)
==837==    by 0x96795FD: AddDocumentCtx_Free (document.c:320)
==837==    by 0x96796AA: doReplyFinish (document.c:179)
==837==    by 0x96796AA: replyCallback (document.c:184)
==837==    by 0x1AA73F: moduleHandleBlockedClients (module.c:3550)
==837==    by 0x139DF6: beforeSleep (server.c:1222)
==837==    by 0x136A0D: aeMain (ae.c:497)
==837==    by 0x13354B: main (server.c:3894)
==837==  Address 0x50005a0 is 32 bytes before a block of size 48 in arena "client"
==837== 
^C837:signal-handler (1552319932) Received SIGINT scheduling shutdown...
837:M 11 Mar 17:58:52.931 # User requested shutdown...
837:M 11 Mar 17:58:52.935 * Saving the final RDB snapshot before exiting.
==837== Conditional jump or move depends on uninitialised value(s)
==837==    at 0x14407C: lzf_compress (lzf_c.c:153)
==837==    by 0x15A885: rdbSaveLzfStringObject.part.1 (rdb.c:319)
==837==    by 0x15AC54: rdbSaveLzfStringObject (rio.h:95)
==837==    by 0x15AC54: rdbSaveRawString (rdb.c:391)
==837==    by 0x1A6C63: RM_SaveStringBuffer (module.c:3088)
==837==    by 0x9677D25: DocTable_RdbSave (doc_table.c:430)
==837==    by 0x969A6A7: IndexSpec_RdbSave (spec.c:1046)
==837==    by 0x15BA9E: rdbSaveObject (rdb.c:780)
==837==    by 0x15C17B: rdbSaveKeyValuePair (rdb.c:824)
==837==    by 0x15C74C: rdbSaveRio (rdb.c:934)
==837==    by 0x15CCFC: rdbSave (rdb.c:1028)
==837==    by 0x13D561: prepareForShutdown (server.c:2579)
==837==    by 0x13D9D8: serverCron (server.c:1000)
==837== 
==837== Conditional jump or move depends on uninitialised value(s)
==837==    at 0x144081: lzf_compress (lzf_c.c:153)
==837==    by 0x15A885: rdbSaveLzfStringObject.part.1 (rdb.c:319)
==837==    by 0x15AC54: rdbSaveLzfStringObject (rio.h:95)
==837==    by 0x15AC54: rdbSaveRawString (rdb.c:391)
==837==    by 0x1A6C63: RM_SaveStringBuffer (module.c:3088)
==837==    by 0x9677D25: DocTable_RdbSave (doc_table.c:430)
==837==    by 0x969A6A7: IndexSpec_RdbSave (spec.c:1046)
==837==    by 0x15BA9E: rdbSaveObject (rdb.c:780)
==837==    by 0x15C17B: rdbSaveKeyValuePair (rdb.c:824)
==837==    by 0x15C74C: rdbSaveRio (rdb.c:934)
==837==    by 0x15CCFC: rdbSave (rdb.c:1028)
==837==    by 0x13D561: prepareForShutdown (server.c:2579)
==837==    by 0x13D9D8: serverCron (server.c:1000)
==837== 
==837== Invalid read of size 8
==837==    at 0x1E078C: je_arena_mapbitsp_read (arena.h:649)
==837==    by 0x1E078C: je_arena_mapbits_get (arena.h:656)
==837==    by 0x1E078C: je_arena_mapbits_binind_get (arena.h:713)
==837==    by 0x1E078C: je_arena_salloc (arena.h:1218)
==837==    by 0x1E078C: je_isalloc (jemalloc_internal.h:876)
==837==    by 0x1E078C: je_malloc_usable_size (jemalloc.c:2483)
==837==    by 0x143C1D: zfree (zmalloc.c:194)
==837==    by 0x7454D06: Buffer_Free (in /home/guy/redislabsmodules/RedisGears/redisgears.so)
==837==    by 0x9677D2F: DocTable_RdbSave (doc_table.c:431)
==837==    by 0x969A6A7: IndexSpec_RdbSave (spec.c:1046)
==837==    by 0x15BA9E: rdbSaveObject (rdb.c:780)
==837==    by 0x15C17B: rdbSaveKeyValuePair (rdb.c:824)
==837==    by 0x15C74C: rdbSaveRio (rdb.c:934)
==837==    by 0x15CCFC: rdbSave (rdb.c:1028)
==837==    by 0x13D561: prepareForShutdown (server.c:2579)
==837==    by 0x13D9D8: serverCron (server.c:1000)
==837==    by 0x136795: processTimeEvents (ae.c:327)
==837==    by 0x136795: aeProcessEvents (ae.c:466)
==837==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==837== 

Docker :edge image broken

With and without PythonHomeDir:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
ImportError: No module named site

Improve performance

  1. Multi threaded execution.
  2. Do not convert record to pyobject, instead expose the record to the python as is using pyobject api.

StreamReader returns records in reverse order (new to old)

127.0.0.1:6379> XADD stream * n 0
"1553045219236-0"
127.0.0.1:6379> XADD stream * n 1
"1553045221269-0"
127.0.0.1:6379> XADD stream * n 2
"1553045223704-0"
127.0.0.1:6379> RG.PYEXECUTE "GearsBuilder('StreamReader').map(lambda x: str(x['n'])).run('stream')"
1) 1) "TotalExecutionDuration"
   2) (integer) 1486200
   3) "TotalReadDuration"
   4) (integer) 1171400
2) 1) "2"
   2) "1"
   3) "0"

Building the docker image has some cpython errors

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _ssl               _tkinter
bsddb185           bz2                dbm
dl                 gdbm               imageop
readline           sunaudiodev        zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
cmath              math

[CRASH] StreamReader appears to be broken

Minimal reproduce (key can either exist or not):

127.0.0.1:6379> RG.PYEXECUTE "GearsBuilder('StreamReader').run('stream')"
Error: Server closed the connection

Trace is stripped (docker) but still:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 18 Mar 2019 22:37:08.005 # Redis 5.0.3 crashed by signal: 11
1:M 18 Mar 2019 22:37:08.005 # Crashed running the instruction at: 0x7f87607e5f99
1:M 18 Mar 2019 22:37:08.005 # Accessing address: 0xa0d4b4f2b1c
1:M 18 Mar 2019 22:37:08.005 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/redisgears.so(+0x5ef99)[0x7f87607e5f99]

Backtrace:
redis-server *:6379(logStackTrace+0x37)[0x55c008ae1d57]
redis-server *:6379(sigsegvHandler+0xb0)[0x55c008ae2480]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f876306c0e0]
/usr/lib/redis/modules/redisgears.so(+0x5ef99)[0x7f87607e5f99]
/usr/lib/redis/modules/redisgears.so(+0x57225)[0x7f87607de225]
/usr/lib/redis/modules/redisgears.so(+0x5767d)[0x7f87607de67d]
/usr/lib/redis/modules/redisgears.so(+0x571c8)[0x7f87607de1c8]
/usr/lib/redis/modules/redisgears.so(+0x5767d)[0x7f87607de67d]
/usr/lib/redis/modules/redisgears.so(+0x571c8)[0x7f87607de1c8]
/usr/lib/redis/modules/redisgears.so(+0x5767d)[0x7f87607de67d]
/usr/lib/redis/modules/redisgears.so(+0x571c8)[0x7f87607de1c8]
/usr/lib/redis/modules/redisgears.so(+0x573cf)[0x7f87607de3cf]
/usr/lib/redis/modules/redisgears.so(+0x57b94)[0x7f87607deb94]
/usr/lib/redis/modules/redisgears.so(+0x58cb8)[0x7f87607dfcb8]
/usr/lib/redis/modules/redisgears.so(+0x67480)[0x7f87607ee480]
/usr/lib/redis/modules/redisgears.so(+0x6776a)[0x7f87607ee76a]
/usr/lib/redis/modules/redisgears.so(+0x67d93)[0x7f87607eed93]
/usr/lib/redis/modules/redisgears.so(event_base_loop+0x2cb)[0x7f87607ef4f3]
/usr/lib/redis/modules/redisgears.so(+0x5674e)[0x7f87607dd74e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f87630624a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8762da4d0f]

------ INFO OUTPUT ------
# Server
redis_version:5.0.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5a396a3a77241301
redis_mode:standalone
os:Linux 4.9.125-linuxkit x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1
run_id:52f368ee763cf2c9026cc7f5221a225445a3ade8
tcp_port:6379
uptime_in_seconds:17
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:9444755
executable:/data/redis-server
config_file:

# Clients
connected_clients:1
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:1

# Memory
used_memory:4320880
used_memory_human:4.12M
used_memory_rss:13090816
used_memory_rss_human:12.48M
used_memory_peak:4320880
used_memory_peak_human:4.12M
used_memory_peak_perc:101.10%
used_memory_overhead:841862
used_memory_startup:792168
used_memory_dataset:3479018
used_memory_dataset_perc:98.59%
allocator_allocated:4518600
allocator_active:4935680
allocator_resident:9285632
total_system_memory:2095869952
total_system_memory_human:1.95G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.09
allocator_frag_bytes:417080
allocator_rss_ratio:1.88
allocator_rss_bytes:4349952
rss_overhead_ratio:1.41
rss_overhead_bytes:3805184
mem_fragmentation_ratio:3.09
mem_fragmentation_bytes:8857952
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:49694
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1552948610
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:187
total_net_output_bytes:34
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:419b77ef7d4ed6a5fefbd39a7e4207cf8cdd9d16
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.050000
used_cpu_user:0.130000
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Commandstats
cmdstat_rg.pyexecute:calls=2,usec=1474,usec_per_call=737.00

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=4 addr=172.17.0.1:37670 fd=18 name= age=12 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=rg.pyexecute

------ REGISTERS ------
1:M 18 Mar 2019 22:37:08.007 #
RAX:00007f875e5f02f0 RBX:0000000000000000
RCX:00007ffc5f5f6b22 RDX:00000a0d4b4f2b28
RDI:00007f875c400000 RSI:00007f87628097a0
RBP:00007f875e229bb0 RSP:00007f875f57f078
R8 :0000000000000310 R9 :00007f875f57ff28
R10:0076809be0000000 R11:0000000000000246
R12:00007f875c400000 R13:00007f8762839200
R14:00007f8762839200 R15:0000000000000003
RIP:00007f87607e5f99 EFL:0000000000010206
CSGSFS:002b000000000033
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f087) -> 00007f875e229b60
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f086) -> 00007f875e229b60
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f085) -> 00007f875c600900
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f084) -> 00007f875c604840
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f083) -> 00007f875f57feb0
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f082) -> 0000000000589b98
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f081) -> 000000005c901d94
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f080) -> 00007f875c604840
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07f) -> 00007f875f57feb0
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07e) -> 0000000000000001
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07d) -> 000000000003508d
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07c) -> 000055c008b8dca9
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07b) -> 0000000000000000
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f07a) -> 00007f875c607118
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f079) -> 00007f875c603338
1:M 18 Mar 2019 22:37:08.007 # (00007f875f57f078) -> 00007f87607de225

------ FAST MEMORY TEST ------
1:M 18 Mar 2019 22:37:08.007 # Bio thread for job type #0 terminated
1:M 18 Mar 2019 22:37:08.008 # Bio thread for job type #1 terminated
1:M 18 Mar 2019 22:37:08.008 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55c008e17000 (2248704 bytes)
*** Preparing to test memory region 55c0095a9000 (405504 bytes)
*** Preparing to test memory region 7f8758000000 (135168 bytes)
*** Preparing to test memory region 7f875c400000 (4194304 bytes)
*** Preparing to test memory region 7f875e07b000 (3145728 bytes)
*** Preparing to test memory region 7f875e580000 (6815744 bytes)
*** Preparing to test memory region 7f875ed82000 (8388608 bytes)
*** Preparing to test memory region 7f875f583000 (8388608 bytes)
*** Preparing to test memory region 7f875fd84000 (8388608 bytes)
*** Preparing to test memory region 7f8760bdb000 (139264 bytes)
*** Preparing to test memory region 7f8760bfe000 (8388608 bytes)
*** Preparing to test memory region 7f87613ff000 (8388608 bytes)
*** Preparing to test memory region 7f8761c00000 (8388608 bytes)
*** Preparing to test memory region 7f8762400000 (8388608 bytes)
*** Preparing to test memory region 7f8763057000 (16384 bytes)
*** Preparing to test memory region 7f8763274000 (16384 bytes)
*** Preparing to test memory region 7f8763ba1000 (32768 bytes)
*** Preparing to test memory region 7f8763bad000 (4096 bytes)

Support making on OSX

...
ld: unknown option: -Bstatic
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [redisgears.so] Error 1

Can't drop execution

127.0.0.1:6379> rg.pyexecute "gearsCtx('test').map(lambda x:str(x)).run('*')"
(error) Flat  Execution with the given name already exists, pleas drop it first.

127.0.0.1:6379> RG.DROPEXECUTION test
(error) execution plan does not exits

127.0.0.1:6379> rg.getresultsblocking test
(error) execution plan does not exits

GearsBuilder(keysOnly = true)

In stead of creating KeysOnlyGB let's have a Single GearsBuilder object, but allow to set in the constructor if you want keysOnly. (which is by default false)

Filter out keys in register()?

# create the builder
builder = GearsBuilder()
# filter events on key:'all_keys'
builder.filter(lambda x: x['key'] != 'all_keys')
# add the keys to 'all_keys' set
builder.map(lambda x: execute('sadd', 'all_keys', x['key']))
# register the execution on key space notification
builder.register()

can we filter out all_keys in register()

Solve deadlock on large data execution

Deadlock happened when message pipe is full. The write operation then block and wait for someone to read from the pipe while holding the global lock. The problem is that the thread who should read from the pipe tries to acquire the global lock, and we get a deadlock.

Solution mode the msg pipe writing to another thread (not the main thread) that does not require acquiring the global lock. Pass messages to this thread using link list.

[FR] Executions should record their origin

E.g., if a client calls RG.PYEXECUTE, DUMPEXECUTIONS/GETEXECUTION should report information like CLIENT LIST, e.g.:

  • Client name
  • Client ip:port
  • Client connection id

If the call is a map reduce, the originating shard connection should be put.

If the call is a registered plan, maybe display something like "INTERNAL"

Cannot build Docker image

I cannot build the Docker image on a Windows or MacOS host.

Here's the output from a Windows host:

\RedisGears> docker build -t redisgears .
...
...
...
Step 4/16 : ADD . /redisgears
 ---> 65d5942cb6b5
Step 5/16 : WORKDIR /redisgears
 ---> Running in 640e5b92fbf3
Removing intermediate container 640e5b92fbf3
 ---> c31dd8ac97a9
Step 6/16 : RUN make clean;
 ---> Running in 7a76e1a2bcdc
rm -f redisgears.so redisgears.a obj/*.o obj/utils/*.o artifacts/release/*.zip artifacts/snapshot/*.zip
Removing intermediate container 7a76e1a2bcdc
 ---> a8d986b12c72
Step 7/16 : RUN make get_deps PYTHON_ENCODING_FLAG=--enable-unicode=ucs4
 ---> Running in 03f23e6f8abc
cd src/deps/cpython;CFLAGS="-fPIC -DREDIS_ALLOC" ./configure --without-pymalloc --enable-unicode=ucs4   ;make
/bin/sh: 1: ./configure: not found
make[1]: *** No targets specified and no makefile found.  Stop.
make[1]: Entering directory '/redisgears/src/deps/cpython'
make[1]: Leaving directory '/redisgears/src/deps/cpython'
Makefile:71: recipe for target 'python' failed
make: *** [python] Error 2
The command '/bin/sh -c make get_deps PYTHON_ENCODING_FLAG=--enable-unicode=ucs4' returned a non-zero code: 2

Traffic generator

We'd need a script that imports the IMDB dataset and a (number of) aggregation queries to assess gears ran correctly on it

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.