Code Monkey home page Code Monkey logo

memcached's Introduction

Static analysis Testing

memcached

Memcached protocol 'wrapper' for Tarantool.

Getting started

Prerequisites

  • Tarantol 1.6.8+ with header files (tarantool && tarantool-dev packages).
  • Cyrus SASL library (with header files)
  • Python >= 3.7 with next packages (for testing only):
    • PyYAML 5+
    • gevent 21+

Installation

Clone this repository and then build it using CMake:

git clone https://github.com/tarantool/memcached.git
cd memcached
git submodule update --init --recursive
cmake . -DCMAKE_BUILD_TYPE=RelWithDebugInfo -DCMAKE_INSTALL_PREFIX=/usr
make
make install

Or use LuaRocks (in this case you'll need libsmall, libsmall-dev, tarantool-dev packages available from our binary repository at http://tarantool.org/dist/master, and system package libsasl2-dev):

luarocks install https://raw.githubusercontent.com/tarantool/memcached/master/rockspecs/memcached-scm-1.rockspec --local

Usage

box.cfg{}
local memcached = require('memcached')
local instance = memcached.create('my_instance', '0.0.0.0:11211')

Now you're set up and ready to go!

How to connect

Install Tarantool package from repository (described here).

Paste the previous example to /etc/tarantool/instances.enabled/memcached.lua and start it with tarantoolctl start memcached.

Then try the following example:

$ printf "set key 0 60 5\r\nvalue\r\n" | nc localhost 11211
STORED
$ printf "get key\r\n" | nc localhost 11211
VALUE key 0 5
value
END
$ printf "set key2 0 60 6\r\nvalue2\r\n" | nc localhost 11211
STORED
$ printf "get key key2\r\n" | nc localhost 11211
VALUE key 0 5
value
VALUE key2 0 6
value2
END

API

  • local memcached = require('memcached') - acquire a library handle
  • local slab = memcached.slab.info() - show information about slab arena and quota.
  • local instance = memcached.create(<name>, <uri>, <opts>) - create a new instance, register it and run
    • name - a string with instance name
    • uri - a string with uri to bind to, for example: 0.0.0.0:11211 (only TCP is supported now)
    • opts - a table with options, the list of possible options is described in the configuration section
  • local instance = instance:cfg(<opts>) - (re)configure an existing instance. opts - a table with options, same as in create
  • local instance = instance:start() - start an instance
  • local instance = instance:stop() - stop an instance
  • local instance = instance:info() - return execution statistics

Configuration

  • readahead - (default) size of readahead buffer for connection. default is box->cfg->readahead
  • expire_enabled - availability of expiration daemon. default is true.
  • expire_items_per_iter - scan count for expiration (tuples processed in one transaction). default is 200.
  • expire_full_scan_time - time required for a full index scan (in seconds). default is 3600
  • verbosity - verbosity of memcached logging. default is 0.
  • flush_enabled - flush command availability. default is true
  • protocol - the protocol, one of negotiation, binary or text.
    • negotiation - detect the protocol automatically at handshake (the default)
    • binary - binary memcached protocol
    • text - text memcached protocol
  • engine - the engine to store data in
    • memory - store everything in memory. (using memtx engine)
    • disk - store everything on hdd/ssd (using vinyl engine) (not yet supported)
  • space_name - custom name for a memcached space, default is __mc_<instance name>
  • if_not_exists - do not throw error if an instance already exists.
  • sasl - enable or disable SASL support (disabled by default)

SASL support

Usual rules for memcached are applicable for this plugin:

  1. Create user (NOTE: it'll use custom folder):

    echo testpass | saslpasswd2 -p -c testuser -f /tmp/test-tarantool-memcached.sasldb
  2. Place configuration file /etc/sasl2/tarantool-memcached.conf. For example:

    mech_list: plain cram-md5
    log_level: 7
    sasldb_path: /tmp/test-tarantool-memcached.sasldb
    

    NOTE: This will disable 'ANONYMOUS' (and other, that aren't listed) authentication plugins.

    NOTE: This will set logging level to the highest possible

    NOTE: This will set custom path for database path

  3. Run Tarantool with memcached plugin with SASL enabled

    local memcached = require('memcached')
    local instance = memcached.create('my_instance', '0.0.0.0:11211', {
      sasl = true
    })
  4. Use your favorite binary(!!) memcached client, that supports(!!) SASL:

    Example using Python's 'python-binary-memcached' library

    import bmemcached
    client = bmemcached.Client(('127.0.0.1:11211', ), 'testuser', 'testpasswd')
    client.set('key', 'value')
    print(client.get('key'))
    

For custom configuration file path, please, use SASL_CONF_PATH environment variable.

What's supported, what's not and other features

Everything is supported, unless the opposite is stated explicitly

  • Text protocol commands:
    • set/add/cas/replace/append/prepend commands (set section)
    • get/gets commands (including multiget)
    • delete command
    • incr/decr commands
    • flush/version/quit commands
    • verbosity - partially, logging is not very good.
    • stat - reset is supported and all stats too.
  • Binary protocol's commands:
    • get/getk/getq/getkq commands (get section)
    • add/addq/replace/replaceq/set/setq commands (set section)
    • quit/quitq/flush/flushq/noop/version commands
    • gat/gatq/touch/gatk/gatkq commands
    • append/prepend/incr/decr
    • verbosity - partially, logging is not very good.
    • stat - reset is supported and all stats too.
    • SASL authentication is supported
    • range operations are not supported as well.
  • Expiration is supported
  • Flush is supported
  • The protocol is synchronous
  • Full support of Tarantool means of consistency (write-ahead logs, snapshots, replication)
  • You can access data from Lua
  • for now LRU is not supported
  • TAP is not supported (for now)
  • VBucket is not supported (for now)
  • UDP/UNIX sockets are not supported (for now)

Caution

This rock is in early beta.

Hacking

We're collecting information for the module developers in the HACKING.md file.

memcached's People

Contributors

0x501d avatar amdrozdov avatar artemreyt avatar bigbes avatar georgykirichenko avatar kostja avatar leonidvas avatar ligurio avatar nshy avatar opomuc avatar rtsisyk avatar totktonada avatar viciious avatar ylobankov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

memcached's Issues

Using PHP connector leads to a tarantool crash

<?php
// php5 or php7
$memcacheD = new Memcached;   //$memcacheD = new Memcache;   // Both drivers cause the same effects
$memcacheD->addServer('192.168.56.1', 11211);
$memcacheD->set('k1', 'v1'); //  $x = $memcacheD->get('k1');   //   Any command leads to crash
tnt_memcached | Segmentation fault
tnt_memcached | Current time: 1458824739
tnt_memcached | Please file a bug at http://github.com/tarantool/tarantool/issues
tnt_memcached | Attempting backtrace... Note: since the server has already crashed, 
tnt_memcached | this may fail as well
tnt_memcached | #0  0x427cc0 in sig_fatal_cb(int)+112
tnt_memcached | #1  0x7f51232ef340 in ?
tnt_memcached | #2  0x7f512185f965 in ?
tnt_memcached | #3  0x7f512185fcb3 in ?
tnt_memcached | #4  0x7f5121861b86 in ?
tnt_memcached | #5  0x4e1974 in lj_vm_ffi_call+132
tnt_memcached | #6  0x4ffc8e in lj_ccall_func+942
tnt_memcached | #7  0x4dd818 in lj_cf_ffi_meta___call+56
tnt_memcached | #8  0x4df9d7 in lj_BC_FUNCC+52

PVS-Studio fixes

memcached/internal/memcached.c:340 err V695 Range intersections are possible within conditional expressions. Example: if (A < 5) { ... } else if (A < 2) { ... }. Check lines: 338, 340.
memcached/internal/proto_bin.c:122 err V517 The use of 'if (A) {...} else if (A) {...}' pattern was detected. There is a probability of logical error presence. Check lines: 122, 126.
memcached/internal/proto_bin.c:534 err V547 Expression is always true. Probably the '&&' operator should be used here.
memcached/internal/proto_bin.c:1104 warn V547 Expression 'errstr' is always true.

Tarantool memcached doesn't start expire-fiber in docker

Hi!
When I start tarantool with memcached module in docker I have got a message:

2020-01-15 06:56:45.388 [1] main/101/tarantool-entrypoint.lua expiration.c:114 E> Can't start the expire fiber
2020-01-15 06:56:45.388 [1] main/101/tarantool-entrypoint.lua expiration.c:115 E> fio: No such file or directory
2020-01-15 06:56:45.389 [1] main/101/tarantool-entrypoint.lua expiration.c:114 E> Can't start the expire fiber
2020-01-15 06:56:45.390 [1] main/101/tarantool-entrypoint.lua expiration.c:115 E> fio: No such file or directory

For launch I used next settings:

#!/usr/bin/env tarantool
memcached = require('memcached')

box.cfg{
  read_only   = false;

  memtx_memory         = 2 * 1024 * 1024 * 1024;  -- 2Gb
  memtx_min_tuple_size = 36;
  memtx_max_tuple_size = 2 * 1024 * 1024;  -- 2Mb

  wal_mode             = "write";
  wal_max_size         = 256 * 1024 * 1024;  -- 256Mb
  checkpoint_interval  = 60 * 60;
  checkpoint_count     = 10;
  force_recovery       = true;

  log_level    = 5;
  log_nonblock = true;
}

opts = {
    expire_enabled = true;
}

instance = memcached.create('my_instance', '0.0.0.0:11211', opts)

And command:

docker run --name mytesttarantool -p3301:3301 -v ${PWD}:/opt/tarantool tarantool/tarantool tarantool /opt/tarantool/app.lua

if I start the app inside the container:

docker exec -it mytesttarantool sh
tarantool /opt/tarantool/app.lua

Expire fiber starts OK:

2020-01-15 07:01:57.269 [100] main/112/__mc_my_instance_expire I> Memcached expire fiber started

But if I try to watch fiber.info even after several hours I always see the same result:

  113:
    csw: 0
    backtrace:
    - C: '#0  0 in +67'
    fid: 113
    memory:
      total: 516432
      used: 0
    name: __mc_my_instance_expire

It seems, that fiber doesn't work at all.

test.sh: event.h: No such file or directory

copying selected object files to avoid basename conflicts...
  CXXLD    clients/memcapable
  CXX      clients/clients_memslap-memslap.o
  CXX      clients/clients_memslap-generator.o
  CXX      clients/clients_memslap-execute.o
  CXXLD    clients/memslap
  CC       clients/memaslap.o
In file included from clients/ms_thread.h:31:0,
                 from clients/memaslap.c:33:
clients/ms_conn.h:16:19: fatal error: event.h: No such file or directory
 #include <event.h>
                   ^
compilation terminated.
make[4]: *** [clients/memaslap.o] Error 1

https://travis-ci.org/tarantool/memcached/jobs/205763200

Tarantool 1.9 support

Hi,
I was install a memcached module and have this at startup tarantool:

2018-05-07 14:17:46.884 [10871] main/101/memcached.lua F> ./memcached.lua:49: loop or previous error loading module 'memcached'
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> tuple_delete(0x15b2500)
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> vy_tuple_delete(0x15b2500)
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> cpipe_flush_cb: locking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> cpipe_flush_cb: unlocking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] wal/101/main D> cbus_endpoint_fetch: locking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] wal/101/main D> cbus_endpoint_fetch: unlocking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] wal/101/main D> cpipe_destroy: locking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] wal/101/main D> cpipe_destroy: unlocking &endpoint->mutex
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> cbus_destroy: locking &bus->mutex
2018-05-07 14:17:46.884 [10871] main/101/memcached.lua D> cbus_destroy: unlocking &bus->mutex

This module have a support Tarantool 1.9?

Thanks!

Failed to build in alpine 3.5, tarantool 1.8

In file included from /rocks/memcached/memcached/internal/msgpuck.c:34:0:
/usr/include/msgpuck.h:127:2: error: #error MP_SOURCE is not supported anymore, please link libmsgpuck.a
 #error MP_SOURCE is not supported anymore, please link libmsgpuck.a

Touch isn't supported in text protocol

Touch

The "touch" command is used to update the expiration time of an existing item
without fetching it.

touch [noreply]\r\n

  • is the key of the item the client wishes the server to touch

  • is expiration time. Works the same as with the update commands
    (set/add/etc). This replaces the existing expiration time. If an existing
    item were to expire in 10 seconds, but then was touched with an
    expiration time of "20", the item would then expire in 20 seconds.

  • "noreply" optional parameter instructs the server to not send the
    reply. See the note in Storage commands regarding malformed
    requests.

The response line to this command can be one of:

  • "TOUCHED\r\n" to indicate success

  • "NOT_FOUND\r\n" to indicate that the item with this key was not
    found.

https://github.com/memcached/memcached/blob/master/doc/protocol.txt

test: sasl/binary-sasl.test.py fails

I'm going to disable the test in the scope of #49. It however need to be investigated and enabled back. Observed the following errors:

# cat test/var/binary-sasl.result
#---------------------------# sasl tests #--------------------------#
Error on line sasl/binary-sasl.test.py:97: 32 not equal 0
Error on line sasl/binary-sasl.test.py:53: 32 not equal 1
Error on line sasl/binary-sasl.test.py:102: 32 not equal 0
Error on line sasl/binary-sasl.test.py:45: -1 not equal 0
Error on line sasl/binary-sasl.test.py:107: 32 not equal 0
Error on line sasl/binary-sasl.test.py:53: 32 not equal 1
Error on line sasl/binary-sasl.test.py:112: 32 not equal 0
Error on line sasl/binary-sasl.test.py:45: -1 not equal 0
Error on line sasl/binary-sasl.test.py:53: 32 not equal 1
Error on line sasl/binary-sasl.test.py:137: 32 not equal 0
Error on line sasl/binary-sasl.test.py:139: 32 not equal 0
<<--------------------------------------------------
get foo

After that the test stuck.

If we'll install sasl2-bin package in test.sh (for saslpasswd2 tool, see test/sasl/sasl.lua) the errors will be gone, but the test still stucks:

# cat test/var/binary-sasl.result 
#---------------------------# sasl tests #--------------------------#
<<--------------------------------------------------
get foo

Option "expire_items_per_iter" does not work

  1. I set expire_items_per_iter=1000 and run
$echo "flush_all" | nc 127.0.0.1 11211
OK

Tarantool removes 50 records per second.
2. I found a typo in init.lua script:

--- a/memcached/init.lua
+++ b/memcached/init.lua
@@ -236,7 +236,7 @@ local stat_table = {
 local conf_table = {
     readahead            = ffi.C.MEMCACHED_OPT_READAHEAD,
     expire_enabled       = ffi.C.MEMCACHED_OPT_EXPIRE_ENABLED,
-    expire_items_per_item= ffi.C.MEMCACHED_OPT_EXPIRE_COUNT,
+    expire_items_per_iter = ffi.C.MEMCACHED_OPT_EXPIRE_COUNT,
     expire_full_scan_time= ffi.C.MEMCACHED_OPT_EXPIRE_TIME,
     verbosity            = ffi.C.MEMCACHED_OPT_VERBOSITY,
     protocol             = ffi.C.MEMCACHED_OPT_PROTOCOL

After fixing tarantool uses 100% CPU and do not remove anything.

Once created instance can't be created after restart

First time

vagrant@vagrant-ubuntu-trusty-64:~$ tarantool
tarantool: version 1.6.7-398-g1c49334
type 'help' for interactive help
tarantool> box.cfg{slab_alloc_arena = 0.1}
2015-11-09 12:51:14.760 [9442] main/101/interactive C> version 1.6.7-398-g1c49334
2015-11-09 12:51:14.761 [9442] main/101/interactive C> log level 5
2015-11-09 12:51:14.761 [9442] main/101/interactive I> mapping 107374184 bytes for tuple arena...
2015-11-09 12:51:14.763 [9442] main/101/interactive I> initializing an empty data directory
2015-11-09 12:51:14.768 [9442] snapshot/101/main I> creating `./00000000000000000000.snap.inprogress'
2015-11-09 12:51:14.769 [9442] snapshot/101/main I> saving snapshot `./00000000000000000000.snap.inprogress'
2015-11-09 12:51:14.769 [9442] snapshot/101/main I> done
2015-11-09 12:51:14.772 [9442] main/101/interactive I> ready to accept requests

---
...

tarantool> memcached = require('memcached')

---
...

tarantool> memcached.create('my_instance', '0.0.0.0:11211')
2015-11-09 12:51:19.951 [9442] wal/101/main I> creating `./00000000000000000000.xlog.inprogress'
2015-11-09 12:51:19.952 [9442] main/104/my_instance_memcached_expire I> Memcached expire fiber started
2015-11-09 12:51:19.953 [9442] main/105/server/0.0.0.0:11211 I> started

---
- name: my_instance
  uri: 0.0.0.0:11211
  listener:
    fd: 12
    name:
      host: 0.0.0.0
      family: AF_INET
      type: SOCK_STREAM
      protocol: tcp
      port: 11211
  status: r
  service: 'cdata<struct memcached_service *>: 0x7f2ee0821030'
  space:
    index:
      0: &0
        unique: true
        parts:
        - type: STR
          fieldno: 1
        id: 0
        space_id: 512
        name: primary
        type: HASH
      primary: *0
    on_replace: 'function: 0x40aed528'
    temporary: false
    id: 512
    engine: memtx
    enabled: true
    name: __mc_my_instance
    field_count: 0
  opts:
    readahead: 16320
    uri: 0.0.0.0:11211
    expire_enabled: true
    expire_items_per_iter: 200
    verbosity: 0
    expire_full_scan_time: 3600
    name: memcached
...

then exit tarantool and run again

vagrant@vagrant-ubuntu-trusty-64:~$ tarantool
tarantool: version 1.6.7-398-g1c49334
type 'help' for interactive help
tarantool> box.cfg{slab_alloc_arena = 0.1}
2015-11-09 12:52:49.940 [9453] main/101/interactive C> version 1.6.7-398-g1c49334
2015-11-09 12:52:49.941 [9453] main/101/interactive C> log level 5
2015-11-09 12:52:49.942 [9453] main/101/interactive I> mapping 107374184 bytes for tuple arena...
2015-11-09 12:52:49.943 [9453] main/101/interactive I> recovery start
2015-11-09 12:52:49.943 [9453] main/101/interactive I> recovering from `./00000000000000000000.snap'
2015-11-09 12:52:49.945 [9453] main/101/interactive I> recover from `./00000000000000000000.xlog'
2015-11-09 12:52:49.946 [9453] main/101/interactive I> done `./00000000000000000000.xlog'
2015-11-09 12:52:49.946 [9453] main/102/hot_standby I> recover from `./00000000000000000000.xlog'
2015-11-09 12:52:49.947 [9453] main/101/interactive I> done `./00000000000000000000.xlog'
2015-11-09 12:52:49.947 [9453] main/101/interactive I> ready to accept requests

---
...

tarantool> memcached = require('memcached')

---
...

tarantool> memcached.create('my_instance', '0.0.0.0:11211')

---
- error: '/usr/share/tarantool/memcached/init.lua:260: Space with name ''__mc_my_instance''
    is already created'
...

Failed when requiring memcached

Vagrantfile

Installed with luarocks.

When trying step by step Usage instructions from readme (with small change) and got error:

vagrant@vagrant-ubuntu-trusty-64:~$ tarantool
tarantool: version 1.6.6-269-gfe1519e
type 'help' for interactive help
tarantool> box.cfg{slab_alloc_arena = 0.1}
2015-11-02 20:54:24.379 [27246] main/101/interactive C> version 1.6.6-269-gfe1519e
2015-11-02 20:54:24.379 [27246] main/101/interactive C> log level 5
2015-11-02 20:54:24.379 [27246] main/101/interactive I> mapping 107374184 bytes for tuple arena...
2015-11-02 20:54:24.382 [27246] main/101/interactive I> initializing an empty data directory
2015-11-02 20:54:24.389 [27246] snapshot/101/main I> creating `./00000000000000000000.snap.inprogress'
2015-11-02 20:54:24.390 [27246] snapshot/101/main I> saving snapshot `./00000000000000000000.snap.inprogress'
2015-11-02 20:54:24.391 [27246] snapshot/101/main I> done
2015-11-02 20:54:24.395 [27246] main/101/interactive I> ready to accept requests

---
...

tarantool> local memcached = require('memcached')

---
- error: '/home/vagrant/.luarocks/share/lua/5.1/memcached/init.lua:11: bad argument
    #1 to ''load'' (string expected, got nil)'
...

tarantool> 

11: ffi.load(package.searchpath('memcached.internal', package.cpath), true)

Segfault if used with tarantool debug build

Got segfault on both master and this patchset (#54) on tarantool 2.5.0-62-gbb7c3d167 (debug) when run tests (cd test && ./test-run.py).

#0  0x00007f3f837665f1 in raise () from /lib64/libc.so.6
#1  0x00007f3f8375053b in abort () from /lib64/libc.so.6
#2  0x000056508e4b15b7 in sig_fatal_cb (signo=11, siginfo=0x7f3f6cb7f3f0, context=0x7f3f6cb7f2c0) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/main.cc:301
#3  <signal handler called>
#4  0x00007f3f84767eab in slab_from_ptr (ptr=0x7f3f810300a0, slab_mask=0) at /home/alex/p/tarantool-meta/memcached/third_party/small/small/slab_cache.h:203
#5  0x00007f3f847680ce in mempool_free (pool=0x56508feebb50, ptr=0x7f3f810300a0) at /home/alex/p/tarantool-meta/memcached/third_party/small/small/mempool.h:274
#6  0x00007f3f84768416 in iobuf_delete (ibuf=0x7f3f810300a0, obuf=0x7f3f81040480) at /home/alex/p/tarantool-meta/memcached/memcached/internal/network.c:69
#7  0x00007f3f8476acf8 in memcached_handler (p=0x56508feec040, fd=22) at /home/alex/p/tarantool-meta/memcached/memcached/internal/memcached.c:258
#8  0x000056508e675c6b in lj_vm_ffi_call () at buildvm_x86.dasc:2578
#9  0x000056508e69bfe9 in lj_ccall_func (L=L@entry=0x4149e370, cd=<optimized out>) at lj_ccall.c:1150
#10 0x000056508e6716c8 in lj_cf_ffi_meta___call (L=0x4149e370) at lib_ffi.c:230
#11 0x000056508e673cbb in lj_BC_FUNCC () at buildvm_x86.dasc:809
#12 0x000056508e650a56 in lua_pcall (L=0x4149e370, nargs=<optimized out>, nresults=-1, errfunc=<optimized out>) at lj_api.c:1158
#13 0x000056508e5e4822 in luaT_call (L=0x4149e370, nargs=3, nreturns=-1) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lua/utils.c:1053
#14 0x000056508e5dd150 in lua_fiber_run_f (ap=0x7f3f81001470) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lua/fiber.c:443
#15 0x000056508e4b0e69 in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=0x56508e5dd0dc <lua_fiber_run_f>, ap=0x7f3f81001470)
    at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lib/core/fiber.h:782
#16 0x000056508e606a71 in fiber_loop (data=0x0) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lib/core/fiber.c:869
#17 0x000056508e7493b4 in coro_init () at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/third_party/coro/coro.c:110

#54 (comment)

Failed to make on clean machine

I'm trying to install on clean vagrant box (Vagrantfile).

At first I installed some dependencies which is not listed in readme.

And then I had an error:

==> default: ---
==> default: Configuration summary for libmemcached version 1.1.0
==> default:    * Installation prefix:       /usr/local
==> default:    * System type:               unknown-linux-gnu
==> default:    * Host CPU:                  x86_64
==> default:    * C Compiler:                cc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
==> default:    * C Flags:                   -g -O2  -Wno-unknown-pragmas -Wno-pragmas -Wall -Wextra -Wjump-misses-init -Wno-attributes -Waddress -Wvarargs -Warray-bounds -Wbad-function-cast -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat-security -Wformat=2 -Wformat-y2k -Wlogical-op -Wmaybe-uninitialized -Wmissing-field-initializers -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wnormalized=id -Woverride-init -Wpointer-arith -Wpointer-sign -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-overflow=1 -Wswitch-enum -Wtrampolines -Wundef -Wunsafe-loop-optimizations -funsafe-loop-optimizations -Wclobbered -Wunused -Wunused-result -Wunused-variable -Wunused-parameter -Wunused-local-typedefs -Wwrite-strings -fwrapv -pipe -fPIE -pie -Wsizeof-pointer-memaccess -Wpacked
==> default:    * C++ Compiler:              c++ (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
==> default:    * C++ Flags:                 -g -O2 -Wno-unknown-pragmas -Wno-pragmas -Wall -Wextra -Wno-attributes -Wvarargs -Waddress -Warray-bounds -Wchar-subscripts -Wcomment -Wctor-dtor-privacy -Wfloat-equal -Wformat=2 -Wformat-y2k -Wmaybe-uninitialized -Wmissing-field-initializers -Wlogical-op -Wnon-virtual-dtor -Wnormalized=id -Woverloaded-virtual -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-overflow=1 -Wswitch-enum -Wtrampolines -Wundef -Wunsafe-loop-optimizations -funsafe-loop-optimizations -Wc++11-compat -Wclobbered -Wunused -Wunused-result -Wunused-variable -Wunused-parameter -Wunused-local-typedefs -Wwrite-strings -Wformat-security -fwrapv -pipe -fPIE -pie -Wsizeof-pointer-memaccess -Wpacked
==> default:    * CPP Flags:                  -fvisibility=hidden
==> default:    * LIB Flags:                 
==> default:    * Assertions enabled:        no
==> default:    * Debug enabled:             no
==> default:    * Shared:                    no
==> default:    * Warnings as failure:       no
==> default:    * SASL support:              yes
==> default:    * make -j:                   no
==> default:    * VCS checkout:              no
==> default: 
==> default: ---
==> default: [  0%] 
==> default: Built target libmemcached_cfg
==> default: Scanning dependencies of target libmemcached_make
==> default:   GEN      libmemcached/csl/scanner.cc
==> default:   CXX      clients/clients_memcapable-memcapable.o
==> default:   CXX      libmemcached/clients_memcapable-byteorder.o
==> default:   CXX      clients/utilities.lo
==> default:   CXXLD    clients/libutilities.la
==> default:   CXX      libmemcached/csl/libmemcached_libmemcached_la-context.lo
==> default: In file included from libmemcached/csl/context.cc:39:0:
==> default: ./libmemcached/csl/context.h:41:37: fatal error: libmemcached/csl/parser.h: No such file or directory
==> default:  #include "libmemcached/csl/parser.h"
==> default:                                      ^
==> default: compilation terminated.
==> default: make[3]: 
==> default: *** [libmemcached/csl/libmemcached_libmemcached_la-context.lo] Error 1
==> default: make[2]: 
==> default: *** [CMakeFiles/libmemcached_make] Error 2
==> default: make[1]: 
==> default: *** [CMakeFiles/libmemcached_make.dir/all] Error 2
==> default: make: 
==> default: *** [all] Error 2

SASL Authentication Mechanism

Overview

Most deployments of memcached today exist within trusted networks where clients may freely connect to any server and the servers don't discriminate against them. There are cases, however, where memcached is deployed in untrusted networks or where administrators would like to exercise a bit more control over the clients that are connecting. This page mostly exists to describe the protocol.

Authentication Concepts

Authentication is abstracted from the server using the Simple Authentication and Security Layer. Among other things, this provides administrators with consistent credential management that is mostly independent from the services that are authenticating clients.

Protocol Definitions

Error Codes and Conditions

There are two status codes provided by the SASL protocol to enable authentication:

Unauthorized

If a message is returned with a status code of 0x20, this is considered an authentication or authorization failure. This may be in response to an explicit authentication command indicating the credentials were not accepted or the authorization was otherwise not granted to access the server.

Continue Authentication

Some SASL mechanisms require multiple messages to be sent between the client and server. If a server responds to an authentication message with a status code of 0x21, this will indicate your client needs to do more work to complete the authentication negotiation.

Authentication Not Supported

If a server responds to an authentication request indicating the command is unknown (status 0x81), it likely doesn't support authentication. It is generally acceptable for the client to consider authentication successful when communicating to a server that doesn't support authentication.

Authentication Requests

List Mechanisms

In order to negotiate authentication, a client may need to ask the server what authentication mechanisms it supports. A command 0x20 with no extras, key, or value will request a mechanism list from the server. The mechanisms are returned as a space-separated value.

Authentication Request

To begin an authentication request, send a request with command 0x21, the requested mechanism as the key, and the initial authentication data as the value if any is required for the chosen mechanism.

Authentication Continuation

If the authentication request responded with a continuation request (status 0x21), the body will contain the data needed for computing the next value in the authentication negotiation. The next step's data will be transmitted similarly to the initial step, but using command 0x22. Note that this includes the mechanism within the key as in the initial request.

Error Reference

+-------------+------------------------------------------+
| Status Code | Meaning                                  |
+-------------+------------------------------------------+
| 0x20        | Authentication required / Not Successful |
| 0x21        | Further authentication steps required.   |
+-------------+------------------------------------------+

Command Reference

+----------------------+------+-----------+-----------+
| Command              | Opc  | Key       | Value     |
+----------------------+------+-----------+-----------+
| List Mechanisms      | 0x20 | None      | None      |
| Start Authentication | 0x21 | Mechanism | Auth Data |
| Authentication Step  | 0x22 | Mechanism | Auth Data |
+----------------------+------+-----------+-----------+

SASL Howto

Introduction

In order to use memcached in a hostile network (e.g. a cloudy ISP where the infrastructure is shared and you can't control it), you're going to want some kind of way to keep people from messing with your cache servers.

SASL (as described in RFC2222) is a standard for adding authentication mechanisms to protocols in a way that is protocol independent.

Getting Started

In order to deploy memcached with SASL, you'll need two things:

  1. A memcached server with SASL support (version 1.4.3 or greater built with --enable-sasl)
  2. A client that supports SASL

Configuring SASL

For the most part, you just do the normal SASL admin stuff.

# Create a user for memcached.
saslpasswd2 -a memcached -c cacheuser

Running Memcached

In order to enable SASL support in the server you must use the -S flag.

The -S flag does a few things things:

  1. Enable all of the SASL commands.
  2. Require binary protocol only.
  3. Require authentication to have been successful before commands may be issued on a connection.

P.S.

Original: https://code.google.com/p/memcached/wiki/SASLAuthProtocol
Original: https://code.google.com/p/memcached/wiki/SASLHowto
Motive to duplicate - google source code (with project wikis) is closing.

Plan

  • SASL_LIST_MECHS
  • SASL_AUTH
  • SASL_STEP

Memcached range operation

Overview

Several implementations of servers speaking the memcached protocol have backends that allow for inexpensive operations over ranges of keys. The purpose of this document is to standardize the protocol for making use of these operations.

General Properties of Range Commands

Commands that take ranges all follow the same pattern of key representation. Each range representation will have a minimum and maximum range as well as flags to indicate whether each end is inclusive or exclusive. Either end may be null indicating there's no limit on the side where the null appears. Keys should be treated as byte arrays for range comparisons. The implementation is not expected to perform any type of character conversion or collation.

Abstract Representation of Ranges

  • null :: means that left/right border is end/begin respectively;
  • [/] :: means that left/right border respectively is included;
  • (/) :: means that left/right border respectively is not included;

Binary Protocol Generalities

Request

In the binary protocol, we've got some extra space and can apply that to defining ranges. The "key" field will be used as the start key. A key length of zero represents null as an empty key can't be stored. The extra info will contain the following info:

+-------------+------+
| Fields      | Size |
+-------------+------+
| End key len | 16   |
| Reserved    | 8    |
| Flags       | 8    |
| Max results | 32   |
+-------------+------+

Flags is a bitmask indicating whether the ends are inclusive or exclusive.

  • (flags & 1) :: If non-zero, start key is inclusive.
  • (flags & 2) :: If non-zero, end key is inclusive.

Max results represents the number of objects the operation is permitted to affect. A value of zero indicates there should be no limit.

A server may enforce a maximum limit and respond with an error if the requested limit exceeds the limit enforced by the server (or is zero).

The end key immediately follows the maximum results.

Response

Each requests will generally have many responses and will return similarly to the way stats works in the binary protocol. The final response will be one with an empty key.
The binary protocol allows for commands to operate in a "quiet mode" where most responses are quelled. No such facility is available for the text protocol as it can't be done consistently without introducing new verbs (and then can't be done safely).

Text Protocol Generalities

The general form of a range command is as follows (please see details for how this varies by command):

CMD <start inclusion> <end inclusion> <max items> <start key> [end key]\r\n

The end key is optional. If not specified, it is considered null. Null is not explicitly included for the start key as there is a natural minimum of keys which can be used in its place.

The inclusion flags are either 0 or 1. If 0, the endpoint is not included. If 1, the endpoint is included.

This is different from the rget as specified in memcachedb to allow for infinite ranges.

Supported commands

get

Ranged get operations allow multiple key/value pairs returned for a single key request.

set

Ranged set operations allow for a range of existing items to be set to a specific value.

For example, if you have stats stored in a server with keys prefixed as stats. you can reset all of them in a single operation by setting (stats., stats/) to "0". This range specifies everything that begins with "stats." excluding "stats." itself.

Upon batch mutation, each modified value will receive a new distinct CAS identifier.

Binary Protocol Details

Each modified value will respond as a plain set does although the key will be included in the response. The final response will contain no key.

In the case of a quiet request, no response will be given unless the command fails.

Text Protocol Details

The command for text set is rset and has the following form:

rset <start inclusion> <end inclusion> <max items> <flags> <exptime> <bytes> <start key> [end key]\r\n
<data>\r\n

The response will be similar to a gets, but with no values included. Example:

VALUE <key> <flags> 0 [<cas>]\r\n
\r\n

append

Ranged append will append a value to every item within a range.

For both protocols, the response will be in the form of their respective protocol-specific response in ranged set.

Text Protocol Details

The command for text append has the following structure:

rappend <start inclusion> <end inclusion> <max items> <bytes> <start key> [end key]\r\n
<data>\r\n

prepend

Prepend follows the same pattern of append.

delete

Ranged delete will remove all values in the specified range.

For both protocols, the response will be in the form of their respective protocol-specific response in ranged set.

incr

Ranged incr will increment all existing keys within a range by the same amount. This follows the same patterns as existing increment with the obvious exception that it will not initialize keys since it's only modifing things that already exist.

Binary Protocol Details

The extra data includes the standard incr/decr extra data after the ranged extra data.

The responses will include a key and value for every modified item.

Text Protocol Details

The text command for range increment is rincr and is in the following form:

rincr <start inclusion> <end inclusion> <max items> <value> <start key> [end key]\r\n

Responses are the same as a multi-gets.

decr

Ranged decr follows ranged incr.

Command Reference

The following table is a quick reference to all of the commands defined herein. Commands omitted from the text column are intentional as they are not defined outside of the binary protocol.

+-----------+-------+-----------+
| Command   | Bin   | Text      |
+-----------+-------+-----------+
| Get       | 0x30  | rget      |
| Set       | 0x31  | rset      |
| QSet      | 0x32  |           |
| Append    | 0x33  | rappend   |
| QAppend   | 0x34  |           |
| Prepend   | 0x35  | rprepend  |
| QPrepend  | 0x36  |           |
| Delete    | 0x37  | rdelete   |
| QDelete   | 0x38  |           |
| Incr      | 0x39  | rincr     |
| QIncr     | 0x3a  |           |
| Decr      | 0x3b  | rdecr     |
| QDecr     | 0x3c  |           |
+-----------+-------+-----------+

Isolation

Isolation levels are specifically left unspecified. Operations concurrently working on the same data set may interleave, mutually exclude each other, or just stomp all over each other.

Implementors are encouraged to document their intentions.

P.S.

Original: https://code.google.com/p/memcached/wiki/RangeOps
Motive to duplicate - google source code (with project wikis) is closing.

Plan

  • RGET
  • RSET
  • QRSET
  • RAPPEND, RPREPEND
  • QRAPPEND, QRPREPEND
  • RDELETE
  • QRDELETE
  • RINCR, RDECR
  • QRINCR, QRDECR

ci: verify the module on different tarantool versions

I propose to add 1.10, 2.1, 2.2, 2.3, 2.4 and 2.5. We can use the example of test.sh file that allows to enable any of those repositories.

We also can use the new way to enable the repository: curl -L https://tarantool.io/installer.sh | VER=2.5 sudo bash (see here) to save typing. It is acceptable for CI.

Possible memleak

Victor Luchits, [22 ะดะตะบ. 2016 ะณ., 13:58]:
c49af59#diff-cbf39d1ef8e3eb0cf0cfa6bf1d6cea8eR213

con.sasl_ctx  = (struct sasl_ctx *)calloc(1, sizeof(struct sasl_ctx));

ั ะฝะต ะฝะฐัˆั‘ะป ะฝะธะณะดะต free ะดะปั ัั‚ะพะน ัˆั‚ัƒะบะธ

LUA API

Give user ability to work with memcached data from LUA like it's local MC (don't give out internal information about data) and give full compatibility with network binary API.

cmd_set stats are always zero

Try building a Docker image from https://github.com/tarantool/cloud/tree/master/docker/tarantool-cloud-memcached

And use the following script to write to it:

#!/usr/bin/env python

from pymemcache.client.hash import HashClient
import time

client = HashClient([
    ('localhost', 11211)
])


c = int(client.get('some_key') or "0")
while True:
    try:
        client.set('some_key', str(c))
        result = client.get('some_key')

        print "Result: ", result
    except Exception as ex:
        print "Error: ", str(ex)

    time.sleep(1)

    c = c+1

The get_hits will be increased, while the set_hits will remain zero.

Parallel increment on key

Hello.
OS: Centos 6.8 x86_64 with latest updates.
I have 2 node Tarantool 1.7.1.117 with master-master replication for testing and plugin memcached 0.1.0.
When I try at the same time the increment for a single key with two running script on two different masters, the first script updates the key on one master, another script updates the key on another master, I do not get the correct value of the key. It should be 2000000 and get for example 882 392 or 892 956.

PHP script:
$mc = new Memcache; $mc->pconnect("ip_master_1_or_2", 11211); $key = "COUNTER"; $mc->set($key, 1, 0, 7200); for($i=0;$i<1000000;$i++) { $mc->increment($key); }

I try this method with tarantool php client, and everything works correctly, the result: 2000000, and I try this method for Memcached, and everything works correctly, result: 200001.

Changing password without restart

As a person running memcached in production, I want to be able to change passwords without restarting tarantool, which leads to downtime.

assertion raised when using pymemcache without authentication

I enable authentication in the memcached module and then do the following:

from pymemcache.client.base import Client
client = Client(('localhost', 11211))
client.set('some_key', 'some_value')

(note the absence of auth information)

Then I see this:

2016-11-01 10:58:54.566 [1] main/110/server/172.17.0.1:44156 proto_bin.c:1036 E> Wrong magic, closing connection
2016-11-01 10:58:54.566 [1] main/110/server/172.17.0.1:44156 I> Bad magic or exit. Exiting.
Assertion failed: ibuf->wpos >= ibuf->rpos (/usr/include/small/ibuf.h: ibuf_used: 84)

Error in `tarantool binary.lua <running>': malloc(): memory corruption

cd test
./test-run.py --gdb
screen -r
*** Error in `tarantool binary.lua <running>': malloc(): memory corruption: 0x0000000000894f30 ***

Program received signal SIGABRT, Aborted.
0x00007ffff5d85657 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
55      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff5d85657 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff5d86a2a in __GI_abort () at abort.c:89
#2  0x00007ffff5dc3bb3 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff5ebc5f8 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff5dc900e in malloc_printerr (action=1, str=0x7ffff5eb8808 "malloc(): memory corruption", ptr=<optimized out>) at malloc.c:4965
#4  0x00007ffff5dca86f in _int_malloc (av=av@entry=0x7ffff60f0c00 <main_arena>, bytes=bytes@entry=304) at malloc.c:3441
#5  0x00007ffff5dcd4e5 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3213
#6  0x00007ffff51c3d34 in memcached_create (name=0x400e9ed0 "memcached", sid=512)
    at /mnt/data/data/Projects/MRU/tarantool/memcached/memcached/internal/memcached.c:230
#7  0x00000000004e9864 in lj_vm_ffi_call () at buildvm_x86.dasc:2575
#8  0x0000000000508675 in lj_ccall_func (L=L@entry=0x40000378, cd=<optimized out>) at lj_ccall.c:962
#9  0x00000000004e5726 in lj_cf_ffi_meta___call (L=0x40000378) at lib_ffi.c:230
#10 0x00000000004e78c7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
#11 0x000000000050e8a0 in lj_cf_dofile (L=0x40000378) at lib_base.c:423
#12 0x00000000004e78c7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
#13 0x00000000004d3921 in lua_pcall (L=L@entry=0x40000378, nargs=<optimized out>, nresults=nresults@entry=0, errfunc=errfunc@entry=0) at lj_api.c:1055
#14 0x00000000004ab755 in lbox_call (L=L@entry=0x40000378, nargs=<optimized out>, nreturns=nreturns@entry=0)
    at /home/travis/build/tarantool/tarantool/build/root/deb-debian-stretch/tarantool-1.6.8.499/src/lua/utils.c:936
#15 0x00000000004a6639 in run_script_f (ap=<error reading variable: value has been optimized out>)
    at /home/travis/build/tarantool/tarantool/build/root/deb-debian-stretch/tarantool-1.6.8.499/src/lua/init.c:497
#16 0x0000000000425b8c in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
    at /home/travis/build/tarantool/tarantool/build/root/deb-debian-stretch/tarantool-1.6.8.499/src/fiber.h:527
#17 0x00000000004b5930 in fiber_loop (data=<optimized out>)
    at /home/travis/build/tarantool/tarantool/build/root/deb-debian-stretch/tarantool-1.6.8.499/src/fiber.c:462
#18 0x0000000000575f4f in coro_init () at /home/travis/build/tarantool/tarantool/build/root/deb-debian-stretch/tarantool-1.6.8.499/third_party/coro/coro.c:110

ER_NO_TRANSACTION on tarantool-2.2+

When trying to do get:

2020-04-28 16:20:15.736 [54892] main/119/server/127.0.0.1:42002 D> ClientError at /build/usr/src/debug/tarantool-2.2.3.1/src/box/txn.c:788

It is ER_NO_TRANSACTION.

Before splitting transaction and fiber memory regions, box_txn_alloc() may be called w/o active transaction.

Reported by Andrey Yanda.

Many sets/adds start fail on allocation

I'm trying to test eviction feature of ordinary memcached. For that purpose I created tarantool instance with memcached interface with small amount of memory available (~ 2mb or less) and started this small perl script to constantly set a key and get it back to check if it was written correctly:

use strict;
use warnings;

use Test::More 'no_plan';
use Cache::Memcached::Fast;

my $memd = new Cache::Memcached::Fast({
    servers => [ "host:port" ],
    nowait => 0,
    hash_namespace => 0,
    utf8 => ($^V ge v5.8.1 ? 1 : 0),
});

for my $i (8 .. 100_000) {
    my $ttl = 5*60;
    my $store_key   = "ttladdmanythiskey$i";
    my $store_value = "ttladdmanythisvalue$i";

    is(  $memd->get($store_key), undef,              "Get not existent value for iteration $i"  );
    ok(  $memd->set($store_key, $store_value, $ttl), "Set with ttl ok for iteration $i"         );
    is(  $memd->get($store_key), $store_value,       "Cmp value for iteration $i"               );
}

I expect that set will be OK always, regardless of how much space is already used and whether or not the same key was already set (so set() is always ok). But a get errors for set() on ~ 100th (well, randomly, btw) iteration of the loop.

The same error is repoduced when ->add is used instead of ->set.

Also I used tcpdump to see tarantool responses. Some of them were:

        0x0000:  4500 007a 9464 4000 3a06 d7e7 0aff da85  E..z.d@.:.......
        0x0010:  053d e970 2bcb b6a1 d007 31d2 ba66 9ae9  .=.p+.....1..f..
        0x0020:  8018 7120 6ac1 0000 0101 080a 5d23 a5ab  ..q.j.......]#..
        0x0030:  f039 1d5f 5345 5256 4552 5f45 5252 4f52  .9._SERVER_ERROR
        0x0040:  2046 6169 6c65 6420 746f 2061 6c6c 6f63  .Failed.to.alloc
        0x0050:  6174 6520 3833 2062 7974 6573 2069 6e20  ate.83.bytes.in.
        0x0060:  736c 6162 2061 6c6c 6f63 6174 6f72 2066  slab.allocator.f
        0x0070:  6f72 2074 7570 6c65 0d0a                 or.tuple..

So I think the problem is with memcached's eviction support.

Revive CI

  • Remove EOL distros, add new ones (see .travis.yml in tarantooltarantool).
  • Test on tarantool 1.9 (update Travis CI variables); the primary reason is that we have no packages for Ubuntu Bionic prior to 1.9.
  • Fix deploying buckets list: remove 1_8, add 1_9, 1_10 and 2_0.
  • Any other needed fixes.

assertion in `rec_check_slots`

(gdb) bt
#0  0x00007ffff6489cc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff648d0d8 in __GI_abort () at abort.c:89
#2  0x00007ffff6482b86 in __assert_fail_base (
    fmt=0x7ffff65d3830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x6aae60 "((((GCobj *)(uintptr_t)((tv)->fr.func).gcptr32)))->gch.gct == ~(~8u)", file=file@entry=0x6aac20 "lj_record.c",
    line=line@entry=102,
    function=function@entry=0x6ac820 <__PRETTY_FUNCTION__.5539> "rec_check_slots") at assert.c:92
#3  0x00007ffff6482c32 in __GI___assert_fail (
    assertion=0x6aae60 "((((GCobj *)(uintptr_t)((tv)->fr.func).gcptr32)))->gch.gct == ~(~8u)", file=0x6aac20 "lj_record.c", line=102,
    function=0x6ac820 <__PRETTY_FUNCTION__.5539> "rec_check_slots")
    at assert.c:101
#4  0x0000000000559547 in rec_check_slots (J=0x40000558) at lj_record.c:102
#5  0x0000000000564e64 in lj_record_ins (J=0x40000558) at lj_record.c:1959
#6  0x00000000005267a4 in trace_state (L=0x40104a70, dummy=0x0, ud=0x40000558)
    at lj_trace.c:619
#7  0x0000000000540d7a in lj_vm_cpcall ()
#8  0x0000000000526bc7 in lj_trace_ins (J=0x40000558, pc=0x4005a284)
    at lj_trace.c:678
#9  0x000000000050a16b in lj_dispatch_ins (L=0x40104a70, pc=0x4005a288)
---Type <return> to continue, or q <return> to quit---
    at lj_dispatch.c:424
#10 0x0000000000542375 in lj_vm_inshook ()
#11 0x0000000000515832 in lua_call (L=0x40104a70, nargs=3, nresults=-1)
    at lj_api.c:1037
#12 0x00000000004a5bc4 in lbox_call (L=0x40104a70, nargs=3, nreturns=-1)
    at /home/bigbes/src/tarantool/src/lua/utils.h:524
#13 0x00000000004a66f8 in box_lua_fiber_run(typedef __va_list_tag __va_list_tag *) (ap=0x7ffff5000638) at /home/bigbes/src/tarantool/src/lua/fiber.cc:303
#14 0x00000000004296d0 in fiber_cxx_invoke (
    f=0x4a6613 <box_lua_fiber_run(typedef __va_list_tag __va_list_tag *) at /home/bigbes/src/tarantool/src/lua/fiber.cc:292>, ap=0x7ffff5000638)
    at /home/bigbes/src/tarantool/src/fiber.h:537
#15 0x00000000004b59db in fiber_loop (data=0x0)
    at /home/bigbes/src/tarantool/src/fiber.c:440
#16 0x000000000060aea7 in coro_init ()
    at /home/bigbes/src/tarantool/third_party/coro/coro.c:96

reproducible by

./test-run.py binary/binary-toobig. binary/binary-toobig. binary/binary-toobig. binary/binary-toobig. binary/binary-toobig. ...

Build failing with latest tarantool 1.7

/rocks/memcached/memcached/internal/expiration.c: In function 'memcached_expire_loop':
/rocks/memcached/memcached/internal/expiration.c:68:45: error: 'ITER_ALL' undeclared (first use in this function)
   iter = box_index_iterator(p->space_id, 0, ITER_ALL, key, key_end);
                                             ^~~~~~~~
/rocks/memcached/memcached/internal/expiration.c:68:45: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [memcached/CMakeFiles/internalso.dir/build.make:235: memcached/CMakeFiles/internalso.dir/internal/expiration.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:375: memcached/CMakeFiles/internalso.dir/all] Error 2
make: *** [Makefile:128: all] Error 2

Error: Build error: Failed building.

Add some kind of LRU support

Right now there's no mechanism for eviction, that is mandatory for caching services. Adding LRU is not an easy job since it depends on some kind of internal extensions for memory allocations or new data structures.

Ways to handle:

  1. Capped collections - MongoDB citation "fixed-size collections that support high throughput operations that insert and retrieve documents based on insertion order. Capped collections work in a way similar to circular buffers: once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection."
  2. Some kind of external LRU implementation with changes on every get - heavy solution (simpler - some mutable fields in tuple and change this field on every access)
  3. per slab-level LRU, 100% chance that deletion will provide you new tuple

Also, compaction of memory is needed (triggered or automatic)

can't get stored data

lua:

box.cfg{}
local memcached = require('memcached')
local instance = memcached.create('test', '127.0.0.1:11211')

start log:

2016-04-06 22:33:06.681 [3556] main/101/mc.lua C> version 1.6.8-611-g944d005
2016-04-06 22:33:06.681 [3556] main/101/mc.lua C> log level 5
2016-04-06 22:33:06.681 [3556] main/101/mc.lua I> mapping 1073741824 bytes for tuple arena...
2016-04-06 22:33:06.682 [3556] main/101/mc.lua I> initializing an empty data directory
2016-04-06 22:33:06.685 [3556] snapshot/101/main I> creating './00000000000000000000.snap.inprogress'
2016-04-06 22:33:06.685 [3556] snapshot/101/main I> saving snapshot './00000000000000000000.snap.inprogress'
2016-04-06 22:33:06.685 [3556] snapshot/101/main I> done
2016-04-06 22:33:06.726 [3556] main/101/mc.lua I> ready to accept requests
2016-04-06 22:33:06.727 [3556] wal/101/main I> creating './00000000000000000000.xlog.inprogress'
2016-04-06 22:33:06.728 [3556] main/104/__mc_test_expire I> Memcached expire fiber started
2016-04-06 22:33:06.728 [3556] main/105/server/127.0.0.1:11211 I> started
2016-04-06 22:33:06.728 [3556] main C> entering the event loop

Test from example:

$ printf "set key 0 60 5\r\nvalue\r\n" | nc localhost 11211 ;  printf "get key\r\n" | nc localhost 11211
STORED
END

No data!

log after:

2016-04-06 22:36:30.736 [3945] wal/101/main I> creating `./00000000000000000003.xlog.inprogress'

Inexisting keys are logged on default configuration

Found during benchmarking tarantool-memcached vs memcached, that tarantool-memcached eats 100% of CPU on logging:

...
2016-04-24 08:31:02.782 [29594] main/982/server/52.58.16.52:58199 proto_binary.c:988 E> memcached error 1: Not found
2016-04-24 08:31:03.030 [29594] main/983/server/52.58.16.52:58200 proto_binary.c:988 E> memcached error 1: Not found
...

log_level is default, 5. I think we need log it on debug level.

undefined symbol: mp_type_hint

2016-09-03 12:44:57.139 [789] main/101/trafmag tarantoolctl:436 E> Start failed: /usr/local/share/tarantool/memcached/init.lua:11: /usr/local/lib/tarantool/memcached/internal.so: undefined symbol: mp_type_hint

Caused by tarantool/tarantool@4c4c1cc

Solution: add #define MP_SOURCE 1 before #include <msgpuck.h>.

Multiple test types for memcached

  • memcapable (part of libmemcached-utils)
  • memslap (part of libmemcached-utils)
  • memtier_benchmark (rlabs utility)
  • port back memcached binary connector from test subfolder to tarantool/test-run
  • patch tarantool/test-run to support custom ports (search free port, 'reserve' it and put it into environment)
  • we need our custom memcached test set for our specific needs
  • port memcached text tests from bigbes/memcached (Depends on #2)
  • port back memcached text connector from test subfolder to tarantool/test-run
  • add travis-ci support

Test target build failed on travis (ubuntu-xenial)

Now we have a problem with build libmemcached on travis for the test target on ubuntu-xenial (on ubuntu-trusty - OK).
From build log:

Scanning dependencies of target libmemcached_cfg
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/config.guess'
libtoolize: copying file 'build-aux/config.sub'
libtoolize: copying file 'build-aux/install-sh'
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:17: installing 'build-aux/compile'
configure.ac:27: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."
CMakeFiles/libmemcached_cfg.dir/build.make:57: recipe for target 'CMakeFiles/libmemcached_cfg' failed
make[3]: *** [CMakeFiles/libmemcached_cfg] Error 1
CMakeFiles/Makefile2:109: recipe for target 'CMakeFiles/libmemcached_cfg.dir/all' failed
make[2]: *** [CMakeFiles/libmemcached_cfg.dir/all] Error 2
CMakeFiles/Makefile2:212: recipe for target 'CMakeFiles/libmemcached.dir/rule' failed
make[1]: *** [CMakeFiles/libmemcached.dir/rule] Error 2
Makefile:216: recipe for target 'libmemcached' failed
make: *** [libmemcached] Error 2
The command "if [ "${TARGET}" = "test" ]; then
    ./test.sh;
else
    git clone https://github.com/packpack/packpack.git packpack;
    packpack/packpack;
fi;
" exited with 2. 

Make SASL authentication support optional

Some environments don't require SASL authentication support and it would be nice if they could do without the cyrus-sasl-lib dependency.

Alternatively, the libsasl2 library could be loaded at runtime via dlopen.

TAP mechanism

Overview

TAP provides a mechanism to observe from the outside data changes going on within a memcached server.

Use cases

TAP is a building block for lots of new types of things that would like to react to changes within a memcached server without having to actually modify memcached itself.

Replication

One simple use case is replication.
Upon initial connect, a client can ask for all existing data within a server as well as to be notified as values change.
We receive all data related to each item that's being set, so just replaying that data on another node makes replication an easy exercise.

Observation

Requesting a TAP stream of only future changes makes it very easy to see the types of things that are changing within your memcached instance.

Secondary Layer Cache Invalidation

If you have frontends that are performing their own cache, requesting a TAP stream of future changes is useful for invalidating items stored within this cache.
Ideally, such a stream would not include the actual data that had changed. Today, all TAP streams include full bodies, but specifying new features that can be implemented by engines such as requesting the omission of values is very straightforward.

External Indexing

A TAP stream pointed at an index server (e.g. sphinx or solr) will send all data changes to the index allowing for an always-up-to-date full-text search index of your data.

VBucket Transition

For the purposes of VBucket transfer between nodes, a new type of TAP request can be created that is every item stored in a VBucket (or set of VBuckets) that is both existing and changing, but with the ability to terminate the stream and cut-over ownership of the VBucket once the last item is enqueued.

Protocol

A TAP session begins by initiating a command from the client which tells the server what we're interested in receiving and then the server begins sending /client/ commands back across the connection until the connection is terminated.

Initial Base Message from Client

A TAP stream begins with a binary protocol message with the ID of 0x40.
The packet's key may specify a unique client identifier that can be used to allow reconnects (resumable at the server's discretion).
A simple base message from a client referring to itself as "node1" would appear as follows.

  Byte            0            1            2            3  
 ------+------------+------------+------------+------------
     0         0x80         0x40         0x00         0x05  
     4         0x00         0x00         0x00         0x00  
     8         0x00         0x00         0x00         0x05  
    12         0x00         0x00         0x00         0x00  
    16         0x00         0x00         0x00         0x00  
    20         0x00         0x00         0x00         0x00  
    24   0x6e ('n')   0x6f ('o')   0x64 ('d')   0x65 ('e')  
    28   0x31 ('1')                                         
Field        (offset) (value)
Magic        (0)    : 0x80
Opcode       (1)    : 0x40
Key length   (2,3)  : 0x0005
Extra length (4)    : 0x00
Data type    (5)    : 0x00
Reserved     (6,7)  : 0x0000
Total body   (8-11) : 0x00000005
Opaque       (12-15): 0x00000000
CAS          (16-23): 0x0000000000000000
Extras              : None
Key          (24-29): The textual string: "node1"
Value               : None

Options

Additional TAP options may be specified as a 32-bit flags specifying options. The flags will appear in the "extras" section of the request packet. If omitted, it is assumed that all flags are 0.

Options may or may not have values. For options that do, the values will appear in the body in the order they're defined (LSB -> MSB).

Backfill

BACKFILL (0x01) contains a single 64-bit body that represents the oldest entry (from epoch) you're interested in. Specifying a time in the future (for the server you are connecting to), will cause it to start streaming only current changes.
An example TAP stream request that specifies a backfill of -1 (meaning future only) would look like this:

  Byte            0            1            2            3  
 ------+------------+------------+------------+------------
     0         0x80         0x40         0x00         0x05  
     4         0x00         0x00         0x00         0x00  
     8         0x00         0x00         0x00         0x0c  
    12         0x00         0x00         0x00         0x00  
    16         0x00         0x00         0x00         0x00  
    20         0x00         0x00         0x00         0x00  
    24         0x00         0x00         0x00         0x00  
    28         0x00         0x00         0x00         0x01  
    32   0x6e ('n')   0x6f ('o')   0x64 ('d')   0x65 ('e')  
    36   0x31 ('1')         0x00         0x00         0x00  
    40         0x00         0xff         0xff         0xff  
    44         0xff                                         
Field        (offset) (value)
Magic        (0)    : 0x80
Opcode       (1)    : 0x40
Key length   (2,3)  : 0x0005
Extra length (4)    : 0x08
Data type    (5)    : 0x00
Reserved     (6,7)  : 0x0000
Total body   (8-11) : 0x00000015
Opaque       (12-15): 0x00000000
CAS          (16-23): 0x0000000000000000
Extras       (24-31): 0x0000000000000001
Key          (32-36): The textual string: "node1"
Value               : 0x00000000ffffffff

Dump

DUMP (0x02) contains no extra body and will cause the server to transmit only existing items and disconnect after all of the items have been transmitted.
An example TAP stream request that specifies only dumping existing records would look like this:

  Byte            0            1            2            3  
 ------+------------+------------+------------+------------
     0         0x80         0x40         0x00         0x05  
     4         0x00         0x00         0x00         0x00  
     8         0x00         0x00         0x00         0x0c  
    12         0x00         0x00         0x00         0x00  
    16         0x00         0x00         0x00         0x00  
    20         0x00         0x00         0x00         0x00  
    24   0x6e ('n')   0x6f ('o')   0x64 ('d')   0x65 ('e')  
    28   0x31 ('1')         0x00         0x00         0x00  
    32         0x00         0xff         0xff         0xff  
    36         0xff                                         
Field        (offset) (value)
Magic        (0)    : 0x80
Opcode       (1)    : 0x40
Key length   (2,3)  : 0x0005
Extra length (4)    : 0x08
Data type    (5)    : 0x00
Reserved     (6,7)  : 0x0000
Total body   (8-11) : 0x0000000c
Opaque       (12-15): 0x00000000
CAS          (16-23): 0x0000000000000000
Extras       (24-31): 0x0000000000000002
Key          (32-36): The textual string: "node1"
Value               : None

VBucket List

LIST_BUCKETS (0x04) is used to limit a request to a specific set of VBuckets.

The VBuckets are included as values of 16-bits each, starting with a 16-bit number indicating the number of VBuckets in the list.

An example TAP stream request that specifies VBuckets 1, 2, and 5 would look like this:

  Byte            0            1            2            3  
 ------+------------+------------+------------+------------
     0         0x80         0x40         0x00         0x05  
     4         0x00         0x00         0x00         0x00  
     8         0x00         0x00         0x00         0x0c  
    12         0x00         0x00         0x00         0x00  
    16         0x00         0x00         0x00         0x00  
    20         0x00         0x00         0x00         0x00  
    24   0x6e ('n')   0x6f ('o')   0x64 ('d')   0x65 ('e')  
    28   0x31 ('1')         0x00         0x00         0x00  
    32         0x00         0xff         0xff         0xff  
    36         0xff         0x00         0x03         0x00  
    40         0x01         0x00         0x02         0x00  
    44         0x05                                         
Field        (offset) (value)
Magic        (0)    : 0x80
Opcode       (1)    : 0x40
Key length   (2,3)  : 0x0005
Extra length (4)    : 0x08
Data type    (5)    : 0x00
Reserved     (6,7)  : 0x0000
Total body   (8-11) : 0x0000000c
Opaque       (12-15): 0x00000000
CAS          (16-23): 0x0000000000000000
Extras       (24-31): 0x0000000000000004
Key          (32-36): The textual string: "node1"
Value               : {3, 1, 2, 5} (16-bits each)

Takeover VBuckets

TAKEOVER_VBUCKETS (0x08) is used to indicate that the client wishes to completely take the given VBuckets away from the server.

Support ACK

SUPPORT_ACK (`0x10~) indicates the client supports explicit ACKing. See ACK support for more information on this mechanism.

Request Keys Only

REQUEST_KEYS_ONLY (0x20) does pretty much what you'd think. It requests that the server does not send the values along with the keys. The server is not required to understand this so the client shouldn't assume it will be guaranteed to not receive values.

Response Commands

After initiating TAP, a series of responses will begin streaming commands back to the caller. These commands are similar to, but not necessarily the same as existing commands.

In particular, each command includes a section of engine-specific data as well as a TTL to avoid replication loops.

General Response Flags

The flag section of the packet has two defined flags that may be present for any given packet:

ACK

0x01 indicates the packet requests an ACK. The client must reply to this packet (i.e. send a packet back upstream with the same opaque) to acknowledge receipt of this packet.

NO_VALUE

0x02 indicates the item may have a value, but it is not included in the request.

Mutation

All mutation events arrive as TAP_MUTATION (0x41) events. These are conceptualy similar to set commands.

Delete - 0x42

Flush - 0x43

Opaque

0x44 - Engine specific extensions are sent as TAP opaque messages. A client may ignore any of these it doesn't understand (though must still honor the ACK if it the message has one).

VBucket Set

0x45 - When doing a takeover, this message indicates a VBucket state transition is ready.

ACK Support

In default mode, all events are streamed out of the server effectively blindly and as fast as possible. With ACKs, a client can more reliably receive messages by letting the server know at a higher protocol level that it has successfully processed TAP messages.

TCP, of course, guarantees message delivery, but a message can be delivered to a remote system that can crash before processing it. With ACKs enabled (and if supported by the server), the server can retransmit any messages whose ACKs were pending at the time of a connection drop if the same named connection reattaches to a TAP session that is still live.

The server chooses how frequently to send ACKs, and may dynamically adjust the interval at which it sends ACK requests to achieve maximum throughput. Clients need not make any assumptions about message ACKs other than any message requesting an ACK needs a response.

Implementations

  • spymemcached - supports a pretty rich API in tap in recent versions
  • ep-engine - has a python implementation that's used in a lot of utilities
  • gotap - is a working go implementation of tap, though has no real-world deployments.
  • Trond's libcouchbase - has support for tap.

P.S.

Original: https://code.google.com/p/memcached/wiki/Tap
Motive to duplicate - google source code (with project wikis) is closing.

Plan

No plan yet

Nicer error output

Is pretty cryptic output for 'no value was passed':

proto_bin.c:135 E> problem while parsing package 'SET' with opaque 8126464
proto_bin.c:136 E> erroneous 'val' section
memcached.c:115 E> Server error 20: Invalid arguments
proto_bin.c:1102 E> memcached error 20: Invalid arguments

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.