Code Monkey home page Code Monkey logo

node-tarantool's Introduction

Connector - high-level Tarantool driver

Connector implements Tarantool binary protocol and exposes nice interface to access Tarantool.

Connector uses Transport to compose and parse request and response headers.

NPM

npm install tarantool

Notes on Tarantool Connector

Connector hides protocol-related stuff under the hood but there are things to know about before using it.

Tarantool database consists of Spaces (tables) and Tuples (rows). Spaces have no name, only numeric id.

This module provides three interfaces — Connection, Mapping and Space.

Connection contains methods to send all kinds of requests, but arguments (and results) are binary Tuples — not usable.

Mapping knows how to convert Object to Tuple and vice versa (thanks to spec) and Space is a Mapping with specified space id.

Call ping on Connector, call on Mapping and insert, select, update, delete on Space.

Object-to-Tuple binding specification — spec

There are three inner types in Tarantool storage: int32, int64 and buffer (octet string). All integers (options, flags, fields, space ids) are unsigned.

Connector can encode some pseudo-types:

  • int32: Unsigned 32-bit integer.
  • int53: Stored as internal int64. A native unsigned JS Number limited to 53 bit, stored natively without lost of significance.
  • int64: Accepts and returns BigNum objects from bignum.
  • buffer: Raw binary buffer.
  • string: Stored as buffer UTF-8 string.
  • object: Stored as buffer UTF-8 string with JSON content.

In order to use custom type, instead of providing it's name, pass object having pack: (value) -> buffer and unpack: (buffer) -> value methods as in example below.

Build spec object to map Object and Tuples to each other. Tarantool knows nothing about field names or types and Mapping maps them depending on spec.

Example of valid spec:

spec = id: 'int32', name: 'string', flags: 'object', smth_hard: {pack: ((value) -> ...), unpack: ((buffer) -> ...)}

We specify three field-related things: order, name and type. spec tells Mapping which place should every field take and how to convert it.

Use any string containing '32' to specity int32 type, same for 53 and 64

API

Connection

tc stands for Tarantool Connection

# create Connection using default Transport
# default port is 33013
tc = Tarantool.connect port, host, callback
# OR create Connection using Transport, any object, with `request(type, body, callback)`
# tc = new Tarantool transport

# make use of connection
tc.insert space, tuple, [flags,] callback
tc.select space, tuples, [index, [offset, [limit,]]] callback
tc.update space, tuple, [operations, [flags,]] callback
tc.delete space, tuple, [flags,] callback
tc.call proc, tuple, [flags,] callback
tc.ping callback
  • space, flags, offset and limit are Integers
  • space is Space number
  • flags is optional field, possible values are stored in Tarantool.flags in camelCase, e.g. Tarantool.flags.returnTuple
  • offset and limit are optional, use them to specify ammount of returned with select
  • tuples is an Array of tuples
  • tuple is an Array of Fields, each Field is Buffer
  • proc is a String
  • operations is an Array of operation, they are constructed via Mapping or Space methods (see below)
  • callback is a Function that is called as callback (returnCode, body) where body is an Array of tuples or a Number or an error string if returnCode is non-zero.

Mapping

Use Mapping to access several spaces with similar structure.

Mapping API:

# creating mapping with specified spec
mapping = tc.mapping spec

# forgetting about tuples
mapping.insert space, object, [flags,] callback
mapping.select space, objects, [index, [offset, [limit,]]] callback
mapping.update space, object, [operations, [flags,]] callback
mapping.delete space, object, [flags,] callback
mapping.call proc, object, [flags,] callback

callback will be called as callback (returnCode, body) where body is an Array of Objects or a Number or an error string if returnCode is non-zero.

# creating operations list — see below
mapping.assign argument
mapping.add argument
mapping.and argument
mapping.xor argument
mapping.or argument
mapping.delete argument
mapping.insertBefore argument
mapping.splice spliceArgument

spliceArgument is a Hash (Object) with three keys: string (String), offset (Number) and length (Number).

argument is a Hash (Object) with single key.

Space

Space incapsulates Mapping and space number to give you shortest API:

# creating Space with spec
space = tc.space space, spec
# OR with mapping
# mapping = tc.mapping spec
# space = tc.space space, mapping

space.insert object, [flags,] callback
space.select objects, [index, [offset, [limit,]]], callback
space.update object, [operations, [flags,]] callback
space.delete object, [flags,] callback

# creating operations list
space.assign argument
space.add argument
space.and argument
space.xor argument
space.or argument
space.delete argument
space.insertBefore argument
space.splice spliceArgument

Operations

Tarantool's update deals with "operations" — atomic field actions.

Here's an example:

spec = id: 'i32', name: 'string', winner: 'i32'
userSpace = tc.space 2, spec
operations = [
    userSpace.or winner: 1
    userSpace.splice name: offset: 0, length: 0, string: '[Winner] '
]
userSpace.update { id: userId }, operations, ->
    console.log 'winner now has [Winner] tag before his name'

TODO

  • check if Buffer.concat is fast enough, if it is slow - replace with array of buffers, concat only before transport.request
  • check argument type in operations
  • catch socket errors and reconnect
  • write about Tarantool keys and multi-object select
  • write tests

Bugs and issues

Bug reports and pull requests are welcome.

LICENSE

Tarantool Connector for node.js is published under MIT license.

node-tarantool's People

Contributors

devgru avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

tarantool stinkie

node-tarantool's Issues

2 параллельных запроса конфликтуют между собой

Допустим у нас есть нижеследующий код, задача которого вынуть нужные объекты по уникальному индексу из разных инстансов тарантула.

Ожидается, что первый запрос вернет следующее из 1го space инстанса на 33019 порту

{ '0': 0, '1': [ { user_id: 1, shard_id: 2 } ] }

а второй из 0го space инстанса на 33016 порту

{ '0': 0, '1': [ { id: 1, name: 'valera' } ] }

Собственно код

var tr = require('tarantool');

var cn1 = tr.connect(33019, '127.0.0.1');
var cn2 = tr.connect(33016, '127.0.0.1');

var spec = {
    'user_id':'i32',
    'shard_id':'i32'
}

var spec2 = {
    'id':'i32',
    'name':'string'
}

var space = cn1.space(1, spec);
var space2 = cn2.space(0, spec2);

space.select([{'user_id':1}], function(){
    console.log(arguments);
});

space2.select([{'id':1}], function(){
    console.log(arguments);
});

Вместо этого получаем ошибку

Error: trying to call absent callback #0
  at TarantoolTransport.processResponse (/usr/lib/node_modules/tarantool/node_modules/tarantool-transport/lib/index.js:88:13)
  at TarantoolTransport.dataReceived (/usr/lib/node_modules/tarantool/node_modules/tarantool-transport/lib/index.js:71:12)
  at Socket.<anonymous> (/usr/lib/node_modules/tarantool/node_modules/tarantool-transport/lib/index.js:47:20)
  at Socket.EventEmitter.emit (events.js:95:17)
  at Socket.<anonymous> (_stream_readable.js:746:14)
  at Socket.EventEmitter.emit (events.js:92:17)
  at emitReadable_ (_stream_readable.js:408:10)
  at emitReadable (_stream_readable.js:404:5)
  at readableAddChunk (_stream_readable.js:165:9)
  at Socket.Readable.push (_stream_readable.js:127:10)
  at TCP.onread (net.js:528:21)

И вместо ожидаемого результата

{ '0': 0, '1': [ { id: 1, name: '\u0002\u0000\u0000\u0000' } ] }

И больше ничего

Обращаем внимание что выглядит результат как 2 инта. То есть выглядит это так будто результат 1го запроса был обработан по спеке 2го.

Комментируем сам второй запрос

//space2.select([{'id':1}], function(){
//    console.log(arguments);
//});

Ошибка про коллбэк ушла но результат по прежнему

{ '0': 0, '1': [ { id: 1, name: '\u0002\u0000\u0000\u0000' } ] }

Становится ясно, что результат первого запроса обработан по спеке 2го запроса ибо первый запрос шел по спеке

var spec = {
    'user_id':'i32',
    'shard_id':'i32'
}

А сам второй запрос вообще закомментирован. Но не закоментирован код маппинга

var space2 = cn2.space(0, spec2);

Комментим его и получаем

{ '0': 0, '1': [ { user_id: 1, shard_id: 2 } ] }

ЧЯДНТ?

ReferenceError: args is not defined

ReferenceError: args is not defined
  at Tarantool.call (/var/www/babylon/node_modules/tarantool/lib/index.js:149:27)
  at Mapping.call (/var/www/babylon/node_modules/tarantool/lib/mapping.js:112:27)

index.coffee:129

tuple = compose.tuple args

Меняем на

tuple = compose.tuple tuple

Не могу использовать модуль после установки

После установки из git или версии из npm нода не видит модуль. Пишу на js, но coffee стоит, папки lib или src в папке с модулем нету.

  "main": "./lib",
  "scripts": {
    "prepublish": "coffee --bare --compile --output lib src/*.coffee",

ЧЯДНТ?

user@user-virtual-machine .../tarantool-tests/100m-test $ npm install git+https://github.com/devgru/node-tarantool.git
npm WARN package.json [email protected] No description
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No README data
npm http GET https://registry.npmjs.org/bignum
npm http GET https://registry.npmjs.org/int53
npm http GET https://registry.npmjs.org/tarantool-transport
npm http 304 https://registry.npmjs.org/bignum
npm http 304 https://registry.npmjs.org/int53
npm http 304 https://registry.npmjs.org/tarantool-transport

> [email protected] install .../tarantool-tests/100m-test/node_modules/tarantool/node_modules/bignum
> node-gyp configure build

make: Вход в каталог `.../tarantool-tests/100m-test/node_modules/tarantool/node_modules/bignum/build'
  CXX(target) Release/obj.target/bignum/bignum.o
  SOLINK_MODULE(target) Release/obj.target/bignum.node
  SOLINK_MODULE(target) Release/obj.target/bignum.node: Finished
  COPY Release/bignum.node
make: Выход из каталога `.../tarantool-tests/100m-test/node_modules/tarantool/node_modules/bignum/build'
[email protected] node_modules/tarantool
├── [email protected]
├── [email protected]
└── [email protected]
user@user-virtual-machine ~.../100m-test $ node index.js 

module.js:340
    throw err;
          ^
Error: Cannot find module 'tarantool'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (.../tarantool-tests/100m-test/index.js:1:79)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
user@user-virtual-machine ~.../tarantool-tests/100m-test $ ls node_modules/tarantool/
node_modules  package.json  README.md

varuint32 numbers encoded by 'leb' module are not compatible with Tarantool

The composer code uses 'leb' node module to encode varuint32.
I realized that encoded results produced by this module are not compatible with results produced by Tarantool.

For example, take length=147.
leb.encodeUInt32(147):
<Buffer 93 01>
but should be:
19, 129

http://en.wikipedia.org/wiki/LEB128

Each byte will have the group in its 7 least significant bits. Set the most significant bit on each byte except the last byte.

Tarantool do this in inverted way - the most significant bit is set only on the last byte. Probably it was implemented to be compatible Perl's pack.

If you have some time, I would really appreciate it if you could take a look on C and/or Python versions and fix the driver:
https://github.com/tarantool/tarantool/blob/master/include/pickle.h#L206
https://github.com/mailru/tarantool-python/blob/master/src/tarantool/request.py#L43

I think that the C version can be directly ported into JS using Buffer class.

Thanks.

Cant call function with arguments

localhost> lua function f1(arg) return 'hello'..arg end

jtest.js:

var Tarantool, host, port, tt;

var i32 = 'i32'; var i53 = 'i53'; var i64 = i53;
var fieldsList = {id: i53, f1: i53, f2: i32, f3: i32, f4: i32, f5: i32, f6: i32, f7: i32, f8: i32};

host = '127.0.0.1';
port = 33013;

Tarantool = require('tarantool');
tt = Tarantool.connect(port, host, function() {
return tt.ping(function() { console.log((Date.now() / 1000) + ': Tnt is ok'); });
});

tt.call('f1', [], function(code, data) {
if (data) {
console.log(JSON.stringify(data) + '_');
}
});

Only this works(but without function arguments):
tt.call('f1', [], function(code, data) {

Don't work:
tt.call('f1', {"f1": 1}, function(code, data) {
tt.call('f1', [{"f1": 1}], function(code, data) {
tt.call('f1', [1], function(code, data) {

What's wrong?

tarantool 1.5.1-108 (and js driver near this version date).

Опечатка assign, и нехватка leb

в space.coffee
Line 23 assign: (object) -> @mapping.assing object # @mapping.assiNG на assign

в compose.coffee

Line 40 length = leb.encodeUInt32 operation.argument.length
не подключен leb

и вопрос, как вызвать lua функцию из node.js???

Подсказки при использовании int64

Возможно, следует быть более толерантными к использованию int64 - автоматически оборачивать Number если он попадает в такое поле, это решается на уровне соответствующего default transformer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.