mcavage / node-zkplus Goto Github PK

Hi,

I really like the zkplus . Thanks for such a nice abstraction . I am currently struck at a point. I am able to catch the close event but not able to catch the leader and newLeader event. Below is my code. Please help

election.vote(function (err, isLeader) {
if(err) console.log(err);
console.log(isLeader); // => 'true' || 'false'
});

election.on('leader' , function(){
console.log("new leader--------------->>>>>>>>>>");
});

election.on('newLeader' , function(leader){
console.log("new leader" , leader);
});

election.on('close', function () {
console.log("new leader stopped--------------->>>>>>>>>>");
});

Simple test file: https://gist.github.com/3921228

Launch it, then do multiple ./zkServer.sh restart on your server.

You will then see that children events are growing by a factor of 2 each time your restart your server.

This is due to two things:

• the code

  node-zkplus/lib/client.js

  Line 798 in 733906c

  if (!done) {

• the fact that when a session event occurs, node-zookeeper will trigger it on every damn watch https://github.com/yfinkelstein/node-zookeeper#known-bugs--issues

As there are a lot of session event on Zookeeper, it must be affecting every node zookeeper user or node zkplus user.

I really dont know how to deal with that.

@mcavage ,@yunong,

I see a lot of changes around connection loss recently and I wonder if the refactoring is finished or if we should wait a little bit more before using it.

Will you increment the version number when the refactoring will be finished?

Regards,

Hello, I had a need where I would want to watch for a path that whom I don't know if it's already created or not.

So I need a watch function that will handle this case.

I wrote one, basically It will give you errors when the node does not exists, has been deleted, or value is not JSON compliant.

But if it has a value, you wil be notified each time it changes.

It goes like this (Yes ugly right now):

function onupdate(zkpath, callback) {
  var
    path = require('path'),
    parts = zkpath.split('/'),
    parent = parts.slice(0, parts.length - 1).join('/') || '/',
    child = parts[parts.length - 1];

  zk.on('error', callback);
  zk.watch(parent, {method: 'list', initialData: true}, function(err, listener) {
    if (err !== null) {
      listener.stop();
      return callback(err);
    } else {
      listener.on('error', function(err) {
        listener.stop();
        callback(err)
      });
      listener.on('children', function (children) {
        if (children.indexOf(child) !== -1) {
          zk.watch(zkpath, {initialData: true}, function(err, listener) {
            listener.on('error', function(err) {
              listener.stop();
              callback(err);
            });
            listener.on('data', function(data) {
              callback(null, data);
            });
          });
        } else {
          callback(new Error('Children does not exists atm'));
        }
      });
    }
  });
}

Usage:

onupdate('/mynode', function(err, data) {

});

Do you think it would fit into ZKplus?

I find it very handy to just give a path and be notified each time it changes, without having to specify options, flags, etc..

(Possibly related to Issue #3 from 2012?)

I have a trivial code sample:

var zkplus = require( 'zkplus' );

var client = zkplus.createClient({
          connect: false
        , servers: [{
                  host: 'registry1'
                , port:  2401
          }]
        , timeout: 30000
});

client.on( 'connect', function() {
        console.log( 'DEBUG: connect called' );

        client.readdir('/', function( error, nodes ) {
                if( error ) {
                        console.log( 'Read error: ' + error );
                } else {
                        console.log( nodes.join() );
                }
                client.close();
        });
});

client.on( 'error', function( error ) {
        console.log( 'General error: ' + error );
});

client.connect( function( error ) {
        console.log( 'Connection error: ' + error );
        process.exit( 1 );
});

... which, with ZooKeeper 3.4.5 and nodejs v0.10.32 on Ubuntu 12.04 LTS always outputs:

DEBUG: connect called
DEBUG: connect called
<list of zookeeper nodes>
Read error: ZooKeeperError: zookeeper is closing

If connect is set to 'true' and the client.connect() function-call is commented out, then the onConnect callback never seems to be triggered at all.

Hi,

Because of the issue #3 (connect parameter not taken into account), I fall into another bug.

If the connect function is called twice, 100% of the CPU is used.

Reproduction :

• create a client without specifying connect:false
• call the function connect

I think that it's because the second call to zk.connect is failing.

You could add a call to getState() to be sure that the connection is not already set before trying to create a connection to zookeeper.

Note: I think that the problem I see is this one : yfinkelstein/node-zookeeper#14

Anthony.

Hi, am i right that you are support your lib only for smart os? :) Sorry for this question, but it is critical for me cause i'm using your module in production.

I've noticed the following:

[2014-05-12T21:57:02.753Z] TRACE: manatee-integ-tests/ZKPlus/79866 on fa858d48-4cc5-6cd9-a6b0-9d07f5603265 (/root/node-manatee/node_modules/zkplus/lib/client.js:186 in close): close: entered
[2014-05-12T21:57:02.753Z] TRACE: manatee-integ-tests/ZKPlus/79866 on fa858d48-4cc5-6cd9-a6b0-9d07f5603265 (/root/node-manatee/node_modules/zkplus/lib/client.js:685 in stop): removing all listeners (path=/yunong/election)
[2014-05-12T21:57:02.754Z] TRACE: manatee-integ-tests/ZKPlus/79866 on fa858d48-4cc5-6cd9-a6b0-9d07f5603265 (/root/node-manatee/node_modules/zkplus/lib/client.js:204): close: done
[2014-05-12T21:57:02.754Z] TRACE: manatee-integ-tests/ZKPlus/79866 on fa858d48-4cc5-6cd9-a6b0-9d07f5603265 (/root/node-manatee/node_modules/zkplus/lib/client.js:707 in onChildren): watch: error getting data (path=/yunong/election, rc=-116, message="zookeeper is closing")
[2014-05-12T21:57:02.756Z] FATAL: manatee-integ-tests/79866 on fa858d48-4cc5-6cd9-a6b0-9d07f5603265 (/root/node-manatee/manatee.js:176): watch: error event fired; exiting
    ZooKeeperError: zookeeper is closing
        at onChildren (/root/node-manatee/node_modules/zkplus/lib/client.js:712:55)
XXX 69434ea3-2064-4166-bb14-994f10166586
XXX []
Uncaught watch: unsolicited error event: zookeeper is closing

Basically if the consumer has setup a watch, and then explicitly invokes close() -- say on a client timeout -- the emitter in watch() will still fire an 'error' even after close has successfully returned. Close() removes all of the 'error' listeners from the watch emitter, which causes the uncaught exception.

Does your module support current libuv / node master in case of new libuv?

Elections currently inline the IP addr of the node as part of the znode name. Making this flexible would mean that we can inline other information as well, such as a port number.

npm does not seem to have the latest version of zkplus.

I'd like to understand a bit more about an error that I get every once in a while.

exception: operation timeout
exception type: undefined
exception arguments: undefined
stack: ZooKeeperError: operation timeout
at onChildren

from what I understand, the client could not connect to my zk cluster and get status. But I would like the client to check again, because the host is quite busy and it's possible nodejs simply didn't get to that callback yet.

Under what conditions does the timeout occur? What is the timeout value? Is this the same timeout as specified on createClient? How many times did zkplus try, i.e. are there configurable retries?

thanks in advance!

Sorry to spam you with issues but.

Why not accept all data formats in zkplus?

One thing that blocked me was that I wanted to store some text into /a/node and it failed because I had to wrap it with JSON.

Default data format should be to store object as they are.

If you want to use JSON you should be able to do like nconf do:

The file store is also extensible for multiple file formats, defaulting to JSON. To use a custom format, simply pass a format object to the .use() method. This object must have .parse() and .stringify() methods just like the native JSON object.

I don't think we should default to JSON even if it's the current way of operating of zkplus.
It's not obvious that node-zkplus is working with only-json as when you're using node-zookeeper you just store what you want in the format you want.

var zk = zkplus({
                 servers: [],
                 format: {
                     parse: JSON.parse,
                     stringify: JSON.stringify
                   }
                });

What do you think?

I can do this, but then we should bump to a major version.

Hi,

{connect : false} in the createClient function is turned to true in the function.

Please change the line ( https://github.com/mcavage/node-zkplus/blob/master/lib/index.js#L34):

connect: options.connect || true (which is obviously always true)

to

connect: (options.connect !== undefined ? options.connect : true)

Regards,
Anthony.

Since the commit 0881c8b, the test "connect no-op" fails.

Here some infos :

node v0.8.8
zookeeper 3.4.3

My server is correctly running during the test session.

-> % npm test             

> [email protected] test /Users/anthony/Fasterize/zkplusmcavage
> ./node_modules/.bin/nodeunit ./test/*.test.js


client.test.js
2012-11-27 13:37:17,888:32301:ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.3
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@716: Client environment:host.name=minifast.fr.cedexis.com
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@723: Client environment:os.name=Darwin
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@724: Client environment:os.arch=11.4.2
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@725: Client environment:os.version=Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@733: Client environment:user.name=anthony
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@741: Client environment:user.home=/Users/anthony
2012-11-27 13:37:17,889:32301:ZOO_INFO@log_env@753: Client environment:user.dir=/Users/anthony/Fasterize/zkplusmcavage
2012-11-27 13:37:17,889:32301:ZOO_INFO@zookeeper_init@786: Initiating client connection, host=192.168.56.11:2181 sessionTimeout=1000 watcher=0x100f9636e sessionId=0 sessionPasswd=<null> context=0x1019009c0 flags=0
2012-11-27 13:37:17,889:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,890:32301:ZOO_WARN@zk_uv_cb@377: zookeeper is (re)-connecting
2012-11-27 13:37:17,890:32301:ZOO_INFO@zk_uv_cb@399: (re)-connection completed
2012-11-27 13:37:17,890:32301:ZOO_INFO@check_events@1735: initiated connection to server [192.168.56.11:2181]
2012-11-27 13:37:17,890:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,892:32301:ZOO_INFO@check_events@1783: session establishment complete on server [192.168.56.11:2181], sessionId=0x13b41217c7b0049, negotiated timeout=4000
2012-11-27 13:37:17,892:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,892:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,893:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,894:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,895:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.3
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@716: Client environment:host.name=minifast.fr.cedexis.com
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@723: Client environment:os.name=Darwin
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@724: Client environment:os.arch=11.4.2
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@725: Client environment:os.version=Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@733: Client environment:user.name=anthony
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@741: Client environment:user.home=/Users/anthony
2012-11-27 13:37:17,896:32301:ZOO_INFO@log_env@753: Client environment:user.dir=/Users/anthony/Fasterize/zkplusmcavage
2012-11-27 13:37:17,896:32301:ZOO_INFO@zookeeper_init@786: Initiating client connection, host=192.168.56.11:2181 sessionTimeout=1000 watcher=0x100f9636e sessionId=0 sessionPasswd=<null> context=0x1019009c0 flags=0
2012-11-27 13:37:17,896:32301:ZOO_INFO@yield@361: starting poll and timer with fd, interest
2012-11-27 13:37:17,896:32301:ZOO_ERROR@zookeeper_interest@1565: Exceeded timeout and disconnected from server.
2012-11-27 13:37:17,896:32301:ZOO_WARN@yield@325: yield:zookeeper_interest returned error: -112 - session expired

2012-11-27 13:37:17,896:32301:ZOO_ERROR@yield@327: not restarting poll and timer since session expired

FAILURES: Undone tests (or their setups/teardowns): 
- connect no-op

To fix this, make sure all tests call test.done()
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

Should theses errors be emitted on watcher?

Talking of https://github.com/mcavage/node-zkplus/blob/master/lib/client.js#L776

Can do it if you agree

Hi,

The return code of zookeeper is not correcly treated in the method onChildren

function onChildren(rc2, msg2, children) {
   if (rc !== 0) {

should be

function onChildren(rc2, msg2, children) {
  if (rc2 !== 0) {

Update : pull request : #10

The "Hello world example" at http://mcavage.me/node-zkplus/ is missing a comma before "port" when incoking createClient.

The example in the bundled README.md appears to be correct.

Hello.

Since you switched to your mcavage/node-zookeeper fork, there's no more a 'close' event sent when a zookeeper session expires.

We have been able to reproduce this, very easy:

• Start a zookeeper server in a VirtualBox
• Connect to your zookeeper server from your host machine, with a very low timeout (seems 4000ms is the lowest), using latest node-zkplus
• Stop communication between zookeeper server and your client: sudo ifconfig vboxnet0 down
• Wait 5-6s
• Restart communication: sudo ifconfig vboxnet0 up

What happens:

• all watches are getting -1 etc.. as seen in https://github.com/yfinkelstein/node-zookeeper#known-bugs--issues

Expected behavior:

• a close event should be sent by the native zookeeper lib

Your fork does not send a close event, thus our code around session disconnection and reconnect does not work anymore: 6a97b30

Putting back "official" node-zookeeper works well, because we do receive a close event when session expires.

I guess we could hack

What do you think ?

PS1:
I hope you do monitor your zk connection status from a client side.
Because when session expires (eg: network interruption, garbage collector working for more than zookeeper timeout), you will never know it ONLY if you do watch for -1 state in your watches.

PS2:
We could not understand how is it possible that our nodejs client and zookeeper server were not able to reach each other for 30 seconds (our timeout), did you already have experienced similar issues with node-zookeeper? (your fork, the official)

i put together this locking example by following the zookeeper cookbook. let me know if you think it might be a useful thing to put inside node-zkplus and i can through together a pull request.

https://gist.github.com/3127666