Code Monkey home page Code Monkey logo

node-sonic-channel's Introduction

node-sonic-channel

Test and Build Build and Release NPM Downloads Buy Me A Coffee

Sonic Channel integration for Node. Used in pair with Sonic, the fast, lightweight and schema-less search backend.

Sonic Channel lets you manage your Sonic search index, from your NodeJS code. Query your index and get search results, push entries to your index and pop them programmatically.

🇫🇷 Crafted in Nantes, France.

Who uses it?

Crisp

👋 You use sonic-channel and you want to be listed there? Contact me.

How to install?

Include sonic-channel in your package.json dependencies.

Alternatively, you can run npm install sonic-channel --save.

How to use?

1️⃣ Search channel

1. Create the connection

node-sonic-channel can be instanciated in search mode as such:

var SonicChannelSearch = require("sonic-channel").Search;

var sonicChannelSearch = new SonicChannelSearch({
  host : "::1",            // Or '127.0.0.1' if you are still using IPv4
  port : 1491,             // Default port is '1491'
  auth : "SecretPassword"  // Authentication password (if any)
}).connect({
  connected : function() {
    // Connected handler
    console.info("Sonic Channel succeeded to connect to host (search).");
  },

  disconnected : function() {
    // Disconnected handler
    console.error("Sonic Channel is now disconnected (search).");
  },

  timeout : function() {
    // Timeout handler
    console.error("Sonic Channel connection timed out (search).");
  },

  retrying : function() {
    // Retry handler
    console.error("Trying to reconnect to Sonic Channel (search)...");
  },

  error : function(error) {
    // Failure handler
    console.error("Sonic Channel failed to connect to host (search).", error);
  }
});

2. Query the search index

Use the same sonicChannelSearch instance to query the search index:

sonicChannelSearch.query("messages", "default", "valerian saliou")
  .then(function(results) {
    // Query results come there
  })
  .catch(function(error) {
    // Query errors come there
  });

3. Teardown connection

If you need to teardown an ongoing connection to Sonic, use:

sonicChannelSearch.close()
  .then(function() {
    // Close success handler
  })
  .catch(function(error) {
    // Close errors come there
  });

2️⃣ Ingest channel

1. Create the connection

node-sonic-channel can be instanciated in ingest mode as such:

var SonicChannelIngest = require("sonic-channel").Ingest;

var sonicChannelIngest = new SonicChannelIngest({
  host : "::1",            // Or '127.0.0.1' if you are still using IPv4
  port : 1491,             // Default port is '1491'
  auth : "SecretPassword"  // Authentication password (if any)
}).connect({
  connected : function() {
    // Connected handler
    console.info("Sonic Channel succeeded to connect to host (ingest).");
  },

  disconnected : function() {
    // Disconnected handler
    console.error("Sonic Channel is now disconnected (ingest).");
  },

  timeout : function() {
    // Timeout handler
    console.error("Sonic Channel connection timed out (ingest).");
  },

  retrying : function() {
    // Retry handler
    console.error("Trying to reconnect to Sonic Channel (ingest)...");
  },

  error : function(error) {
    // Failure handler
    console.error("Sonic Channel failed to connect to host (ingest).", error);
  }
});

2. Manage the search index

Use the same sonicChannelIngest instance to push text to the search index:

sonicChannelIngest.push("messages", "default", "conversation:1", "I met Valerian Saliou yesterday. Great fun!")
  .then(function() {
    // Push success handler
  })
  .catch(function(error) {
    // Push errors come there
  });

3. Teardown connection

If you need to teardown an ongoing connection to Sonic, use:

sonicChannelIngest.close()
  .then(function() {
    // Close success handler
  })
  .catch(function(error) {
    // Close errors come there
  });

3️⃣ Control channel

1. Create the connection

node-sonic-channel can be instanciated in control mode as such:

var SonicChannelControl = require("sonic-channel").Control;

var sonicChannelControl = new SonicChannelControl({
  host : "::1",            // Or '127.0.0.1' if you are still using IPv4
  port : 1491,             // Default port is '1491'
  auth : "SecretPassword"  // Authentication password (if any)
}).connect({
  connected : function() {
    // Connected handler
    console.info("Sonic Channel succeeded to connect to host (control).");
  },

  disconnected : function() {
    // Disconnected handler
    console.error("Sonic Channel is now disconnected (control).");
  },

  timeout : function() {
    // Timeout handler
    console.error("Sonic Channel connection timed out (control).");
  },

  retrying : function() {
    // Retry handler
    console.error("Trying to reconnect to Sonic Channel (control)...");
  },

  error : function(error) {
    // Failure handler
    console.error("Sonic Channel failed to connect to host (control).", error);
  }
});

2. Administrate your Sonic server

You may use the same sonicChannelControl instance to administrate your Sonic server.

3. Teardown connection

If you need to teardown an ongoing connection to Sonic, use:

sonicChannelControl.close()
  .then(function() {
    // Close success handler
  })
  .catch(function(error) {
    // Close errors come there
  });

List of channel methods

For details on argument values, see the Sonic Channel Protocol specification.

Search channel

  • sonicChannelSearch.query(collection_id<string>, bucket_id<string>, terms_text<string>, [options{limit<number>, offset<number>, lang<string>}<object>]?) ➡️ Promise(results<object>, error<object>)
  • sonicChannelSearch.suggest(collection_id<string>, bucket_id<string>, word_text<string>, [options{limit<number>}<object>]?) ➡️ Promise(results<object>, error<object>)
  • sonicChannelSearch.list(collection_id<string>, bucket_id<string>, [options{limit<number>, offset<number>}<object>]?) ➡️ Promise(results<object>, error<object>)

Ingest channel

  • sonicChannelIngest.push(collection_id<string>, bucket_id<string>, object_id<string>, text<string>, [options{lang<string>}<object>]?) ➡️ Promise(_, error<object>)
  • sonicChannelIngest.pop(collection_id<string>, bucket_id<string>, object_id<string>, text<string>) ➡️ Promise(count<number>, error<object>)
  • sonicChannelIngest.count<number>(collection_id<string>, [bucket_id<string>]?, [object_id<string>]?) ➡️ Promise(count<number>, error<object>)
  • sonicChannelIngest.flushc(collection_id<string>) ➡️ Promise(count<number>, error<object>)
  • sonicChannelIngest.flushb(collection_id<string>, bucket_id<string>) ➡️ Promise(count<number>, error<object>)
  • sonicChannelIngest.flusho(collection_id<string>, bucket_id<string>, object_id<string>) ➡️ Promise(count<number>, error<object>)

Control channel

  • sonicChannelControl.trigger(action<string>, [data<string>]?) ➡️ Promise(_, error<object>)
  • sonicChannelControl.info() ➡️ Promise(results<object>, error<object>)

What is Sonic?

ℹ️ Wondering what Sonic is? Check out valeriansaliou/sonic.

How is it linked to Sonic?

node-sonic-channel maintains persistent TCP connections to the Sonic network interfaces that are listening on your running Sonic instance. In case node-sonic-channel gets disconnected from Sonic, it will retry to connect once the connection is established again.

You can configure the connection details of your Sonic instance when initializing node-sonic-channel from your code; via the Sonic host and port.

node-sonic-channel's People

Contributors

anbraten avatar spacemeowx2 avatar valeriansaliou avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-sonic-channel's Issues

Adding support for Promises

Hi Valerian! Thanks for creating and sharing Sonic as an open source db, I've just started experimenting with it and I think it's just what I'm looking for!

I was also looking at your Node.js library and I was wondering if you thought about support for Promises (thus enabling async/await)?

I've now added a few helpers to enable that but it would be nice to support it out of the box (or make support easier with utils.promisfy by using error first callbacks)

Let me know what you think! (would love to help & contribute PR's)

Sonic Injest Closes Unexpectedly

I'm using the node.js sonic driver to injest a large (100M+) document collection from MongoDB. After ~60 seconds, the connection automatically drops with the following error:

(INFO) - closing channel thread with traceback: Connection reset by peer (os error 104)

I've tried implementing different timeout lengths on my injest stream without any luck. Is there an automatic timeout setting in node-sonic-channel?

Serverless environment

Are there any suggestions on how to handle usage under something like AWS Lambda. I am assuming the best course is to create and destroy a connection on every request as you otherwise may create many orphaned connections. Would you agree? Is there significant overhead to doing it this way?

In depth example

Could you make an in depth example of how to use this for a blog? I'd like to use this for Hugo but I'm not quite sure how to start.

Getting some TCP buffer overflows

Seeing this sometimes, at scale (using node-sonic-channel latest):

Mar 24 20:52:09 shard-1 sonic[2804]: thread 'sonic-channel-client' panicked at 'buffer overflow', src/channel/handle.rs:136:33
Mar 24 20:52:09 shard-1 sonic[2804]: note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Promisse on generic does not resolve (endless loading)

I just found and kinda fixed a bug that makes absolutely no sense, and that's why I'm opening an issue and not a pull request.

Context:

I was working on this API that was already working, there was no reasonable reason to not, but sonic decide to not work today without any change on code or update on his version.
To be fair I was using a new container to sonic (and change it again a couple of times, never nows...)

Resolution (Work-around):

One promise was not returning proper callback, twice.
On lib/channel/geniric.js line 306 where he got the following methods, I just add a return this "return" on line 311. Tested inject and search, both work.
Them, cleaning the mess made debugging (Some try/catch blocks on client, logs on client and sonic-channel) the search method stopped working.
Added this work-around resolve() on line 318 and fixed for now.

SonicChannelGeneric.prototype._wrapDone = function(fn_operation) {
  var self = this;

  return new Promise(function(resolve, reject) {
    // Wrap operation 'done' callback with a 'Promise'
    return fn_operation.call(self, function(data, error) {  // line 311
      if (error) {
        return reject(error);
      }

      return resolve(data);
    });
    return resolve({some: 'thing'});  // line 318
  });
};

Extra information

Find out some calls for __client related to the _executeOrDefer. Looks important but his value was ways empty and triggering the defer not allowing the __execute. But after my work-around, it's working anyway, not sure why.
It's related to some socket.io connection or something like that, totally independent of the connection I'm using?
Also, would be a little weird because the only value attribution to __client appear to be a chain of functions calling each other where the "head" is never called or callable (once its private)

Ingest Push invalid_format

Hi!

This is my code:

sonicChannelIngest.push('test', 'bucket:1', 'object:1', 'hola mundo', { 'lang': 'spa' })
  .then(function () {
    console.log('ok');
  })
  .catch(function (error) {
    console.log('error', error);
  });

I got this error:

{
  code: 'invalid_format',
  message: 'PUSH <collection> <bucket> <object> "<text>"'
}

If i remove options this works fine

Thanks!

[Help] Errors when ingesting many objects (~2000) in bursts

I have a relatively simple use case and not a lot of data:

  • There's a scheduled task that runs every x hours to index some objects in sonic
  • It should remove any outdated data and re-insert the latest state of the objects
  • The procedure looks something like this:
    // flush the whole collection to get rid of potentially deleted objects
    await ingest.flushc(collection)
    
    // ...
    
    // for all our entities
    for (entity of entities) {
      // remove the object (this is a duplicate of the flushc call, it could be skipped)
      await ingest.flusho(collection, bucket, entity.id)
      // re-insert the object with latest data
      await ingest.push(collection, bucket, entity.id, entity.text)
    }

This looks fine in theory to me but I'm experiencing a problem that a lot of data that is apparently pushed by my application code is missing from the Sonic index. My investigations so far have led to some errors thrown by this node-sonic-channel (see questions below) so I'm opening this issue here.

Question(s) 1

I'm a little lost on how to use the Ingest connection over time.

In the above scenario, I'm iterating over n entries and doing stuff that might take some time. Should I open one connection for the whole procedure? As I understand the connection can be closed by timeout (?) or other reasons? Is there a recipe on how to reconnect and continue such a batch procedure in case the connection is closed? Or is it safer to open and close a connection per ingest.push call?

Question 2

What does this error mean and what should I do to prevent it?

Error: Offline stack is full, cannot stack more operations until Sonic Channel connection is restored (maximum size set to: 500 entries)

Probably comes from here.

Question 3

What does this error mean and what should I do to prevent it?

channel closed

Probably comes from here


All of this seems like a very simple use-case so I'm assuming that I'm doing something very wrong. I'd appreciate some help. Thanks!

Does suggest query support LANG search

When push or query data, I use { LANG: "cmn" }, It's Mandarin Chinese, maybe the same as Simple Chinese

It's works well, testing data:

sonicChannelIngest.push("testing", "area_code", "230102", "黑龙江省哈尔滨市道里区", { LANG: "cmn" })
sonicChannelIngest.push("testing", "area_code", "230103", "黑龙江省哈尔滨市南岗区", { LANG: "cmn" })
sonicChannelIngest.push("testing", "area_code", "230104", "黑龙江省哈尔滨市道外区", { LANG: "cmn" })

# query
sonicChannelSearch.query("testing", "area_code",  "黑龙江省", { LANG: "cmn" })

# result
["230102", "230103", "230104"]

when use suggest:

sonicChannelSearch.suggest("testing", "area_code",  "黑龙江省")

got the error:

Error: { code: 'unexpected_error', message: 'query_error' }

so, does suggest support LANG params query, thanks

Tail files for log streaming

I'd imagine it's pretty normal that most apps write logs to files while they are running.

It's not clear (to me at least) whether I need to write my own code to stream the logs to the sonic-server using a library like this, or if sonic-server can watch a pattern of files?

Ingest bulk

I didn't see an example of this, but is it possible to ingest several elements in one push?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.