Code Monkey home page Code Monkey logo

jschan's Introduction

jschan  Build Status

jschan is a JavaScript port of libchan based around node streams

Status

The jschan API should be stable at this point, but libchan is a developing standard and there may be breaking changes in future.

When used over the standard SPDY transport, jschan is compatible with the current Go reference implementation.

We also have a websocket transport to allow us to run jschan on the browser. This feature is not covered by the spec, and is not compatible with other implementations.

Transports

The jschan API has support for swappable transports, some of which are included and others which need to be installed.

Install

npm install jschan --save

Example

This example exposes a service over SPDY. It is built to be interoperable with the original libchan version rexec.

Server

The server opens up a jschan server to accept new sessions, and then execute the requests that comes through the channel.

'use strict';

var spdy = require('jschan-spdy');
var childProcess = require('child_process');
var server = spdy.server();
server.listen(9323);

function handleReq(req) {
  var child = childProcess.spawn(
    req.Cmd,
    req.Args,
    {
      stdio: [
        'pipe',
        'pipe',
        'pipe'
      ]
    }
  );

  req.Stdin.pipe(child.stdin);
  child.stdout.pipe(req.Stdout);
  child.stderr.pipe(req.Stderr);

  child.on('exit', function(status) {
    req.StatusChan.write({ Status: status });
  });
}

function handleChannel(channel) {
  channel.on('data', handleReq);
}

function handleSession(session) {
  session.on('channel', handleChannel);
}

server.on('session', handleSession);

Client

'use strict';

var usage = process.argv[0] + ' ' + process.argv[1] + ' command <args..>';

if (!process.argv[2]) {
  console.log(usage)
  process.exit(1)
}

var spdy = require('jschan-spdy');
var session = spdy.clientSession({ port: 9323 });
var sender = session.WriteChannel();

var cmd = {
  Args: process.argv.slice(3),
  Cmd: process.argv[2],
  StatusChan: sender.ReadChannel(),
  Stderr: process.stderr,
  Stdout: process.stdout,
  Stdin: process.stdin
};

sender.write(cmd);

cmd.StatusChan.on('data', function(data) {
  sender.end();
  setTimeout(function() {
    console.log('ended with status', data.Status);
    process.exit(data.Status);
  }, 500);
})

What can we write as a message?

You can write:

  • Any plain JS object, string, number, ecc that can be serialized by msgpack5
  • Any channels, created from the Channel interface
  • Any binary node streams, these will automatically be piped to jschan bytestreams, e.g. you can send a fs.createReadStream() as it is. Duplex works too, so you can send a TCP connection, too.
  • objectMode: true streams. If it's a Transform (see through2) then it must be already piped with their source/destination.

What is left out?

  • Your custom objects, we do not want you to go through that route, we are already doing too much on that side with jsChan.

API


Session Interface

A session identifies an exchange of channels between two parties: an initiator and a recipient. Top-level channels can only be created by the initiator in 'write' mode, with WriteChannel().

Channels are unidirectional, but they can be nested (more on that later).

session.WriteChannel()

Creates a Channel in 'write mode', e.g. a streams.Writable. The channel follows the interface defined in Channel Interface. The stream is in objectMode with an highWaterMark of 16.

session.close([callback])

Close the current session, but let any Channel to finish cleanly. Callback is called once all channels have been closed.

session.destroy([callback])

Terminate the current session, forcing to close all the involved channels. Callback is called once all channels have been closed.

Event: 'channel'

function (channel) { }

Emitted each time there is a new Channel. The channel will always be a Readable stream.


Channel Interface

A Channel is a Stream and can be a Readable or Writable depending on which side of the communication you are. A Channel is never a duplex.

In order to send messages through a Channel, you can use standards streams methods. Moreover, you can nest channels by including them in a message, like so:

var chan = session.WriteChannel();
var ret  = chan.ReadChannel();

ret.on('data', function(res) {
  console.log('response', res);
});

chan.write({ returnChannel: ret });

Each channel has two properties to indicate its direction:

  • isReadChannel, is true when you can read from the channel, e.g. chan.pipe(something)
  • isWriteChannel, is true when you can write to the channel, e.g. something.pipe(chan)

channel.ReadChannel()

Returns a nested read channel, this channel will wait for data from the other party.

channel.WriteChannel()

Returns a nested write channel, this channel will buffer data up until is received by the other party. It fully respect backpressure.

channel.BinaryStream()

Returns a nested duplex binary stream. It fully respect backpressure.

channel.destroy([callback])

Close the channel now.


jschan.memorySession()

Returns a session that works only through the current node process memory.

This is an examples that uses the in memory session:

'use strict';

var jschan  = require('jschan');
var session = jschan.memorySession();
var assert  = require('assert');

session.on('channel', function server(chan) {
  // chan is a Readable stream
  chan.on('data', function(msg) {
    var returnChannel  = msg.returnChannel;

    returnChannel.write({ hello: 'world' });
  });
});

function client() {
  // chan is a Writable stream
  var chan = session.WriteChannel();
  var ret  = chan.ReadChannel();
  var called = false;

  ret.on('data', function(res) {
    called = true;
    console.log('response', res);
  });

  chan.write({ returnChannel: ret });

  setTimeout(function() {
    assert(called, 'no response');
  }, 200);
}

client();

jschan.streamSession(readable, writable, opts)

Returns a session that works over any pair of readable and writable streams. This session encodes all messages in msgpack, and sends them over. It can work on top of TCP, websocket or other transports.

streamSession is not compatible with libchan.

Supported options:

  • header: true or false (default true), specifies if we want to prefix every msgPack message with its length. This is not needed if the underlining streams have their own framing.
  • server: true or false (default false), specifies if this is the server component or the client component.

About LibChan

It's most unique characteristic is that it replicates the semantics of go channels across network connections, while allowing for nested channels to be transferred in messages. This would let you to do things like attach a reference to a remote file on an HTTP response, that could be opened on the client side for reading or writing.

The protocol uses SPDY as it's default transport with MSGPACK as it's default serialization format. Both are able to be switched out, with http1+websockets and protobuf fallbacks planned. SPDY is encrypted over TLS by default.

While the RequestResponse pattern is the primary focus, Asynchronous Message Passing is still possible, due to the low level nature of the protocol.

Graft

The Graft project is formed to explore the possibilities of a web where servers and clients are able to communicate freely through a microservices architecture.

"instead of pretending everything is a local function even over the network (which turned out to be a bad idea), what if we did it the other way around? Pretend your components are communicating over a network even when they aren't." Solomon Hykes (of Docker fame) on LibChan - [link]

Find out more about Graft

Contributors

License

MIT

jschan's People

Contributors

adrianrossouw avatar azu avatar edef1c avatar mcollina avatar pelger avatar tamagokun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jschan's Issues

Passing Channels between Sessions

Currently a channel, once created, can be serialized only within the session that has originated it.

This means that I cannot route the messages between the services easily, i.e. I cannot send a channel from one user to another client. Currently, this can be done only through explicit piping.

However, I think it's doable to automatic 'fix' this, so when we send a channel bound to another session, it gets automatically piped. What do you think?

msgpack implementation

According to the internals of libchan, we need a new implementation of msgpack that supports extension points (msgpack changed spec some time ago).
😱

Should it be something I put work on? @pelger @Vertice?

Create channels from the global jsChan object

Currently to create a return channel we use:

var jschan  = require('jschan');
var session = jschan.memorySession();
var chan = session.createWriteChannel();
var ret  = chan.createReadChannel();

chan.write( { hello: 'world, ret: ret } );

However it will be much simpler to have:

var jschan  = require('jschan');
var session = jschan.memorySession();
var chan = session.writeChannel(); // shorter
var ret  = jschan.readChannel(); // simpler, channels are derived from the top-level objects, and then passed through. 

chan.write( { hello: 'world, ret: ret } );

protocol error w/ io.js 2.40

see GraftJS/graft#19

events.js:141
      throw er; // Unhandled 'error' event
            ^
Error: Protocol "http:" not supported. Expected "https:".
    at new ClientRequest (_http_client.js:53:11)
    at Object.exports.request (http.js:31:10)
    at ClientSession.newStream [as _createNewStream] (/Volumes/alien/projects/digs/digs-graft/node_modules/jschan/lib/spdy/client.js:104:18)
    at createChannel (/Volumes/alien/projects/digs/digs-graft/node_modules/jschan/lib/spdy/client.js:143:11)
    at ClientSession.WriteChannel (/Volumes/alien/projects/digs/digs-graft/node_modules/jschan/lib/spdy/client.js:157:10)
    at SPDYClient.Client._write (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/lib/client.js:36:30)
    at doWrite (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:279:12)
    at writeOrBuffer (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:266:5)
    at SPDYClient.Writable.write (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:211:11)
    at Graft.ondata (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_readable.js:572:20)

TCP Transport ?

Hi,

Is there a plan for a TCP transport ?
I guess that it will have the highest throughput on local network, which is where jschan is going to be used anyway.

Rexec example: Golang server does not work with jsChan client

Hello everyone,

Based on an email conversation I had with Matteo, he suggested I open an issue here, as his assumption was that the Node client should work with the Golang server, based on last time he tested it.

Basically, I am testing the Golang libchan rexec server:
https://github.com/docker/libchan/blob/master/examples/rexec/rexec_server/server.go

Against the Node.Js jsChan rexec sample:
https://github.com/GraftJS/jschan/blob/master/examples/rexec/client.js

The versions tested are the ones in master repo as of January 15, 2015.
I had two console windows open, on the same server. One running a compiled golang version of libchan rexec server, waiting on port 9323.
The other console would be for the client:

  • the golang client run with "rexec echo hi" works correctly
  • the node client, run with "node index.js echo hi" does not work correctly

I added a few log lines in the server, and the approximate location where the server stops being responsive is right after:
t, err := tl.AcceptTransport() (line 63, it passes this one)
and before
receiver, err := t.WaitReceiveChannel() (line 71, it doesn't seem to pass this one, no errors thrown)

In other words, the t.WaitReceiveChannel() does not appear to be responsive to jsChan's channel.

My regards to the team, great work,
Silviu

Message queueing patterns

Write up concerns about message queue patterns and post it in libchan issue queue for official feedback

sample application

Build out a simple example app (confer with libchan guys to make sure we are building the right thing)

Backpressure problem in stream session

According to the spec at https://nodejs.org/api/stream.html#stream_readable_read_size_1
Note: once the _read() method is called, it will not be called again until the push method is called.

This causes done() to never be called when ByteStream on remote-side has finished and is sending an { id: x } message. When done() is not called, back pressure is not released in session inStream.

stream/channels.js

ByteStream.prototype._read = function() {
74    var done = this._lastDone;
75    if (done) {
76      this._lastDone = null;
77      done();
78    }
79    return null;
80  };

remove async dependency

Async is not used very much, just for closing down some stuff. Maybe we can do better and remove it.

.travis.yml

I'm not an owner here, can any of you add travis to this repo? I already added the .travis.yml file.

webrtc transport

we discussed wanting to enable graft/jschan over webrtc data channels too. this issue is to collect some of the related concepts.

Unix transport

Let's first let the libchan guys how they want to roll out that in libchan.

port usage.md

Port usage.md to node proto-code, so we can see what it's like

Jschan Compatible with node 4.8.3 ?

like the title, is jschan compatible with node 4.8.3?
if there is already has closed issue related to this, please send me a link here.

Thanks

Automatic object stream piping

Currently, we cannot pass a standard objectMode stream inside a channel with automated piping: we need to create a new channel and pipe.

Is it sound to use that pattern? Plus, solving this issue might be more tricky, because Transform stream are not supported by the automated piping natively, you will need to flag them with something like jschanReadable or jschanWritable.

Any opinions on this?

Performance Analysis

How this is slower than standard streams?
How this is slower than HTTP?

We just need some numbers, and maybe some optimization tricks.

Encoding

It seems the encoding part of libchan is in a flux at the moment: e.g. they want to support msgpack (ref)[https://github.com/docker/libchan/blob/72754f8294ce601ea4a947a635df3252bb009e2d/PROTOCOL.md], but right now it's netstrings only, and they have a pull request that adds msgpack, but it does not remove netstrings (see docker/libchan#37).

It's completely unclear.

end and finish event not firing via SPDY

I am creating an array of fs.createReadableStreams and putting them into a message. This message is then passed to my jschan/graft microservice, which pipes each file it into local fs.createWritableStreams.

Data, end, and finish events fire correctly when I am using the in-memory graftJS, and also when using spdy on localhost with a small file (< 3746 bytes). However, all three of the same events fail to fire when sending larger files over spdy jschan.

Node version is v0.10.35

Sender:

    // Convert filePaths into filestreams that can be sent via graft jschan
    if (msg.filePaths) {
      var fileStreams = {};
      msg.filePaths.forEach(function(filePath) {
        var fileName = path.basename(filePath);
        //var zlibCompress = zlib.createDeflate();
        var fileStream = fs.createReadStream(filePath);

        fileStream.resume();
        fileStream.on('data', function(chunk) {
          console.log('got %d bytes of data', chunk.length);
        });
        fileStream.on('end', function() { console.log('end'); });

        fileStreams[fileName] = fileStream;
      });
      msg.fileStreams = fileStreams;
    }

Receiver:

        // Save remote fileStreams to local temp directory.
        async.map(Object.keys(msg.fileStreams), function _writeStream(fileName, cb) {
          var fileStream = msg.fileStreams[fileName];
          var outPath = path.join(dirPath, fileName);
          var outStream = fs.createWriteStream(outPath);
          console.log(outPath);

          // end event does not fire
          fileStream.resume();
          fileStream.on('data', function(chunk) {
            console.log('got %d bytes of data', chunk.length);
          });
          fileStream.on('end', function() { console.log('end'); });

          // finish event does not fire 
          fileStream.pipe(outStream).on('error', function(err) {
            setImmediate(cb, err);
          }).on('finish', function() {
            setImmediate(cb, null, outPath);
          });

        }, function(err, filePaths) {
          if (err) { cleanupTmp(); throw err; }
          ...
        });

Would really appreciate the help nailing this bug down.

node-spdy

Research and send a feedback email to this list about node-spdy

Echoed embedded channels break pipes

One of the things I've need to do in aetherboard is to have a writechannel for all my messages to the server, and then a read channel that is meant to contain the merged stream of everybody's messages in the order they were received.

Inside the message I send, I have another write channel that contains a series of messages that I will write to even after the initial message is sent. This is actually not that uncommon. think IRC with the ability to see everybody's text as they type.

Unfortunately the message I receive back from the server, is somehow different for me than for all the other people who receive the message, and beyond that, it is also different to my initial writechannel.

Trying to pipe from it, causes some kind of silent error, and unpipes everything, meaning that my client becomes disconnected.

This behaviour can be reproduced with aetherboard by following this issue :
GraftJS/aetherboard#16

I have written some tests for graft in this PR :
GraftJS/graft#13

I'm pretty sure though that the bug is actually going to be found here in jschan though.

encoder.js possible bug location

I've also noticed that the passing around of the original channel is causing the chan.streams to contain circular references in there.

@mcollina

Only first frame sent for large objects.

When sending large strings or objects (more than one frame) over a response channel only the first frame appears to be handles correctly. Workaround for now is to use a byte stream. This issue potentially due to a misconfiguration on node-spdy.

port in memory

Port in-memory execution to node so we can try out our api

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.