Code Monkey home page Code Monkey logo

q-connection's Introduction

Build Status

Asynchronous Remote Objects

This library makes it possible for objects to communicate asynchronously between memory-isolated JavaScript contexts, including pipelining interactions with results. Promises serve as proxies for remote objects.

Q-Connection works in Node and other CommonJS module loaders like Browserify, Mr, and Montage.

This is how it looks:

var Q = require("q");
var Connection = require("q-connection");
var remote = Connection(port, local);

The remote object is a promise for the local object on the other side of the connection. Likewise, the other side of the connection will get a promise for your local object. You are not obliged to provide a local object, depending on which end of the connection is providing a service.

If the remote or local object is not serializable, like functions or objects with methods, the other side will receive a promise but you will have to “send messages” to the promise instead of interacting directly with the remote object. When you invoke a method on a remote object, you get a promise for the result and you can immediately pipeline a method call on the result. This is the secret sauce.

The port is any W3C message port, web worker, or web socket. In the W3C’s infinite wisdom, these do not have a unified API, but Q-Connection will normalize them internally.

// To communicate with objects in a worker
var worker = new Worker("worker.js");
var child = Connection(worker, local);
// Inside a worker, to communicate with the parent
var parent = Connection(this);
// To communicate with a remote object on the other side of
// a web socket
var socket = new WebSocket("ws://example.com");
var remote = Connection(socket, local);
// To communicate with a single frame on the same origin
// (multiple frames will require some handshaking event sources)
var iframe = document.frames[0];
var child = Connection(iframe.contentWindow, local, {
    origin: window.location.origin
})
// To communicate with a parent frame on the same origin
var child = Connection(window, local, {
    origin: window.location.origin
})
// With a message port
var port = new MessagePort();
var near = Connection(port[0]);
var far = Connection(port[1]);

Your local value can be any JavaScript value, but it is most handy for it to be an object that supports an API and cannot be serialized with JSON.

var Q = require("q");
var counter = 0;
var local = {
    "next": function () {
        return counter++;
    }
};

In this case, the local object has a "next" function that returns incremental values. Since the function closes on local state (the counter), it can't be sent to another process.

On the other side of the connection, we can asynchronously call the remote method and receive a promise for the result.

remote.invoke("next")
.then(function (id) {
    console.log("counter at", i);
});

The connection is bi-directional. Although you do not need to provide and use both local and remote values on both sides of a connection, they are available.

You can asynchronously interact with any value using the Q API. This chart shows the analogous operations for interacting with objects synchronously and asynchronously.

synchronous                asynchronous
------------------         -------------------------------
value.foo                  promise.get("foo")
value.foo = value          promise.put("foo", value)
delete value.foo           promise.del("foo")
value.foo(...args)         promise.post("foo", [args])
value.foo(...args)         promise.invoke("foo", ...args)
value(...args)             promise.fapply([args])
value(...args)             promise.fcall(...args)

All of the asynchronous functions return promises for the eventual result. For the asynchronous functions, the value may be any value including local values, local promises, and remote promises.

The benefit to using the asynchronous API when interacting with remote objects is that you can send chains of messages to the promises that the connection makes. That is, you can call the method of a promise that has not yet been resolved, so that message can be immediately sent over the wire to the remote object. This reduces the latency of interaction with remote objects by removing network round-trips.

A chain of dependent operations can be contracted from:

<-client     server->
a..
   ''--..
         ''--..
               ''--..
             ..--''
       ..--''
 ..--''
b..
   ''--..
         ''--..
               ''--..
             ..--''
       ..--''
 ..--''
c..
   ''--..
         ''--..
               ''--..
             ..--''
       ..--''
 ..--''

Down to:

<-client     server->
a..
b..''--..
c..''--..''--..
   ''--..''--..''--..
         ''--..--''..
       ..--''..--''..
 ..--''..--''..--''
 ..--''..--''
 ..--''

Where the dotted lines represent messages traveling through the network horizontally, and through time vertically.

Ports

Q-Connection handles a variety of message ports or channel types. They are all internally converted into a Q Channel. If you are using a message channel that provides a different API than this or a WebWorker, WebSocket, or MessagePort, you can adapt it to any of these interfaces and Q-Connection will handle it.

This is probably the simplest way to create a channel duck-type, assuming that you’ve got a connection instance of the Node variety.

var port = {
    postMessage: function (message) {
        connection.send(message);
    },
    onmessage: null // gets filled in by Q-Connection
};
connection.on("message", function (data) {
    port.onmessage({data: ""})
});
var remote = Connection(port, local);

Here's an example showing adapting socket.io to the message port.

var port = {
  postMessage: function (message) {
    socket.emit("message", message);
  },
  onmessage: null // gets filled in by Q-Connection
};
socket.on("message", function(data) {
  port.onmessage({data: data});
});
var remote = Connection(port, local);

Q Channels

  • get() returns a promise for the next message from the other side of the connection. get may be called any number of times independent of when messages are actually received and each call will get a promise for the next message in sequence.
  • put(message) sends a message to the remote side of the connection.
  • close(reason_opt) indicates that no further messages will be sent.
  • closed a promise that is fulfilled with the reason for closing.

Q-Connection exports an indefinite Queue that supports this API which greatly simplifies the implementation of adapters.

  • get() returns a promise for the next value in order that is put on the queue. get may be called any number of times, regardless of whether the corresponding value is put on the queue before or after the get call.
  • put(value) puts a message on the queue. Any number of messages can be put on the queue, indepent of whether and when the corresponding get is called.
  • close(reason_opt) indicates that no further messages will be put on the queue and that any promises for such messages must be rejected with the given reason.
  • closed a promise that is fulfilled when and if the queue has been closed.

Web Workers and Message Ports

Q-Connection detects ports by their postMessage function.

  • postMessage(message)
  • onmessage(handler(message))

Web Sockets

Q-Connection detects Web Sockets by their send function. It takes the liberty to start the socket and listens for when it opens.

  • send(message)
  • addEventListener(event, handler(event))
  • start()
  • open event
  • close event

Memory

Q-Connection uses an LRU cache of specified size. The default size is infinite, which is horribly leaky. Promises between peers will stick around indefinitely. This can be trimmed to something reasonable with the max option.

var remote = Connection(port, local, {max: 1024});

The least frequently used promises will be collected. If the remote attempts to communicate with a collected promise, the request will be ignored. The minimum working set will vary depending on the load on your service.

To be notified when communication is attempted with a collected promise set the onmessagelost option.

var remote = Connection(port, local, {
    max: 1024,
    onmessagelost: function (message) {
        console.log("Message to unknown promise", message);
    }
});

q-connection's People

Contributors

blai avatar coderunr avatar domenic avatar felixge avatar francoisfrisch avatar gozala avatar johnjbarton avatar kriskowal avatar pchoi avatar spearway avatar stuk avatar thibaultzanini avatar wpk- avatar zarutian avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

q-connection's Issues

Understanding q-comm: why the busy loop?

My first q-comm call was an infinite loop. As far as I can tell, my 'invoke" is calling me and I am calling it. Of course it does not fail on JS stack overflow because of the postMessage.

In trying to sort this out I trying stepping through the /2.iframe example. I kept hitting nextTick so I added logging:
nextTick = function (task) {
console.log("nextTick post to port2");
tail = tail.next = {task: task};
channel.port2.postMessage();
};
To my surprise nextTick seems to be called as fast as the CPU can run. The example does work, but I don't understand why the busy loop is needed. It will make finding my problem even more difficult.

q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
3q.js:61nextTick post to port2
index.html:17index.html: Hello, Child!
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
4q.js:61nextTick post to port2
2q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
3q.js:61nextTick post to port2
2q.js:61nextTick post to port2
index.html:17index.html: Hello, Parent!
q.js:61nextTick post to port2
3q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
iframe.html:13iframe.html: Hello, Parent!
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
index.html:26index.html: Hi
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
2q.js:61nextTick post to port2
iframe.html:22iframe.html: Hi
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2
2q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
q.js:61nextTick post to port2
2q.js:61nextTick post to port2

Serialize reference cycles

At the moment, without Q.master, providing a promise for anything with a promise cycle causes problems. It should be possible to serialize object graphs with reference cycles without difficulty.

Channel close(reason) message ignored

Hi Kris,

thanks for this library : ).

I'm using it to do RPC with a child process via newline terminated JSON. However, sometimes the child might misbehave and output invalid JSON, so I'd like to handle this protocol error by calling close(new Error('Invalid JSON ...')) on my channel. This works, but unfortunately the error is being overwritten with new Error("Can't resolve promise because Connection closed") inside q-connection.

I wanted to send a patch for this, but unfortunately it's not easy because q/queue.close() automatically creates an error by itself when none is given, so q-connection would have to detect this case in order to provide its more useful default error message in absence of a user specified one.

I hope this makes some sense, if not I'll send my partial patch for more clarity.

Unrelated: Would you recommend using q / q-connection v2 for a small project at this point?

Cheers,
Felix

Make the default service object a rejection instead of undefined.

Due to a bug in my code I called
Connection(connection, local, options)
with local === undefined.

The result was a error much later on
"post": function (name, value) {
return object[name].apply(object, value);
},
Where we get 'type error' because 'object' is undefined. Because object was closed over from an argument backtracking it was not straight forward.

I think the API would be better if {} rather than "undefined" meant "don't have a local API"

v0.3.1 - All messages are sent to the last connected client

In v0.3.1, when using WebSockets as port, all messages are sent to the last connected client.

Based in https://github.com/kriskowal/q-comm/blob/master/examples/1.node-to-node/example.js, I wrote this gist (https://gist.github.com/2211009) to test the issue.

When the first client is executed, it receives all its messages correctly. But, after the second connection, all messages are sent to the newest client, although only the correct messages are resolved (propagated to the then method).

Date object returned as promise

It is easier demonstrated with the code:

Server:

var QC = require('q-connection'),
      service = {
        time: function() {
            return new Date();
        }
    };
//...
QC.Connection(port, service);
//...

Client:

service = qConnection(port);
service.invoke("time").done(function(data) {
   // Here data is a promise object instead of Date
   console.log('Time:', data.toString());
});

Client and server communicate over a port object, adapted to operate over net.Socket.

Does this work in browser directly?

Hello @kriskowal ,
I am the member of cdnjs project.
We want to host this library.
But there is a question in q-connection.js.

var Q = require("q");
var LruMap = require("collections/lru-map");
var Map = require("collections/map");
var UUID = require("./lib/uuid");
var adapt = require("./adapt");

Does this work in browser directly?
Thanks for your help!

cdnjs/cdnjs#7870
cdnjs/cdnjs#7895

deferred.notify does not reach the other side of connection

I understand progress is optional, but is the following by design? Would be nice if optional progress also work

// server
Connect(port, {
    hello: function (world) {
      var deferred = Q.defer();
      var count = 0;
      setInterval(function() {
        deferred.notify('Hello ' + world + ' from server: ' + count++);
      }, 1000);
      return deferred.promise;
    },
    foo: function (bar) {
      var deferred = Q.defer();
      setTimeout(function() {
        deferred.resolve('Foo ' + bar + ' from server');
      }, 1000);
      return deferred.promise;
    }
});

// client
var remote = Connect(port, local);
remote.invoke('hello', 'of client').progress(function(greet) {
  // will not work
  console.log(greet); // Never reach here!
});

remote.invoke('foo', 'blah of client').then(function(bar) {
  // will work
  console.log(bar); // will log: 'Foo blah of client from server'
});

Understanding q-comm: the meaning of self-receive

Here is a trace using q-comm debug prefixed with the window file. I don't understand the first 'receive' which is reported in the main window but is content from the main window.

index.html: 36: resolve: L"" {}
index.html: 36: sending: R"" "post" ["hi",["Hello, Child!"]]
index.html: 36: receive: {"type":"send","to":"","from":"2C68A813-ADDD-4A52-8F77-9720AAF6F45C","op":"post","args":["hi",["Hello, Child!"]]}
__________________________???
iframe.html: 95: resolve: L"" {}
iframe.html: 95: sending: R"" "post" ["hi",["Hello, Parent!"]]
index.html: Hello, Child!
index.html: 36: receive: {"type":"send","to":"","from":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","op":"post","args":["hi",["Hello, Parent!"]]}
iframe.html: 95: receive: {"type":"send","to":"","from":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","op":"post","args":["hi",["Hello, Parent!"]]}
index.html: Hello, Parent!
iframe.html: Hello, Parent!
index.html: 36: receive: {"type":"resolve","to":"2C68A813-ADDD-4A52-8F77-9720AAF6F45C","resolution":"Hi"}
index.html: 36: resolve: L"2C68A813-ADDD-4A52-8F77-9720AAF6F45C" "Hi"
iframe.html: 95: receive: {"type":"resolve","to":"2C68A813-ADDD-4A52-8F77-9720AAF6F45C","resolution":"Hi"}
index.html: Hi
index.html: 36: receive: {"type":"resolve","to":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","resolution":"Hi"}
iframe.html: 95: receive: {"type":"resolve","to":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","resolution":"Hi"}
iframe.html: 95: resolve: L"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64" "Hi"
iframe.html: Hi
index.html: 36: receive: {"type":"resolve","to":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","resolution":"Hi"}
iframe.html: 95: receive: {"type":"resolve","to":"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64","resolution":"Hi"}
iframe.html: 95: resolve: L"B654DB40-5F8A-40E7-8CDE-D0DE7130AC64" "Hi"

How about this?

Just off the top of my head, if we have this object exposed over q-connection:

{ foo: { bar: 1, baz: [2, 3, 4] } }

why can't we have this syntax to get both foo.bar and foo.baz[1]?

client
  .get(["foo", "bar"], ["foo", "baz", 1])
  .spread(function (a, b) {
    assert a === 1;
    assert b === 3;
  });

Probably could work just as well for calling multiple functions at the same time, etc.

Web Worker in the browser

Hi,

I'm trying to get q-connection to work in the browser with a Web Worker. I have previously got q-comm working in my environment, in which I use it to invoke a remote method and it works quite well, but wanted to move to q-connection.

I have used browserify on q-connection to create a 'standalone' using -s flag to get a single file and have exported 'Connection' but it doesn't seem to work. I keep getting 'global is not defined', because I think that browserify uses the window object to set the global. So I assigned self to a window variable and it sort of works but now I get a Type Error. My code looks like this

//on client
var worker = new Worker('worker.js');

        var remote = Connection(worker,{
            "hi": function (message) {
                console.log(message);
            }
        });
        remote.invoke("hi", "Hello, Child!")
        .then(function (message) {
            console.log(message);
        });

//worker.js
self.window = self; //assign self to window
importScripts('q-conn.js');

var remote = Connection(this, {
    hi: function (message) {
        return message;
        }
    });
remote.invoke("hi", "Hello, Parent!");

Any guidance would be great.

Thanks
Frank

NaN

I suppose JSON is responsible for translating NaN to null?

server:

var remote = Connection(socket, {
  double: function(x){
    var res = x * 2;
    console.log(res);  // --> NaN
    return res;
  }
});

client:

var remote = Connection(socket);
remote.invoke('double', {}).then(function(res){
  console.log(res);  // --> null
});

Since Q-Conn has the communication API, and because this does not occur in local promises (Q(NaN).then(...) // --> NaN) is there a way around this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.