Binary Payload Format about wamp-proto HOT 14 CLOSED

Is now think the crucial decision to make is about application payloads (we have control over WAMP protocol definition, but not application payloads).

Today, application payloads can use anything that can be represented in JSON. We need to maintain that.

The core problem is:

1. JSON => Bencode: the latter lacks the primitive types null, boolean and decimal
2. Bencode => JSON: the latter lacks any binary type

We need to support converting between both without loosing information (roundtripping).

We could quite easily add the missing primitive types to Bencode. The problem is:

• it breaks existing Bencode implementations
• it does not solve 2.

from wamp-proto.

Another question to answer:

Do we want to allow mixed text and binary WAMP messages within 1 session?

from wamp-proto.

Maybe consider other serialization formats (instead, or besides: pluggable serialization):

• http://msgpack.org/
• https://github.com/binaryjs/js-binarypack

from wamp-proto.

MessagePack has no string type - it does not differentiate strings and byte sequences. Neither does Bencode. However, see BinaryPack and js-binarypack.

from wamp-proto.

Another option is BSON. Major benefits:

• automatically supports MongoDB (definitely cool)
• officially supported
• lots of implementations already exist
• almost 1:1 with JSON, so no protocol redefinition
• allows extensions

from wamp-proto.

Yes, is also a candidate. My impression is that BSON is more optimized for access speed, less storage/wire space efficiency. Regarding WAMP itself, we try to impose minimal requirements. The main problem is I think how to handle application paylods (RPC arguments/return types and event payloads): how to map payloads serialized using different formats between WAMP peers?

from wamp-proto.

What about a binary version of HTTP chunked encoding? Pick some format (BSON or whatever), but payloads are chunked in the rest of the packet:

HEADER | PAYLOAD1 | PAYLOAD2 ...

And each PAYLOADX looks like:

ID | LENGTH | DATA

Where:

• ID is single byte (between 1 and 127); MSB reserved for possible later extension beyond 127 payloads
• LENGTH is the length of this chunk (maybe something like the WebSocket frame protocol for length?)
• HEADER is the translation of WAMP to a binary format, without values
  • Payload values are given an ID; 0 represents null
• PAYLOADX is a chunked value

This would allow multiple payloads to be interleaved with minimal overhead, so an RPC call could return multiple payloads with a single request.

from wamp-proto.

Not sure if I get that.

Assume I'd want to return the following single values from a remote procedures called via WAMP:

   "bar"|string

   ".. some binary .. "|binary

    ["bar"|string, ".. some binary .. "|binary]

   ["bar"|string, ".. some binary .. "|binary, 1]

where ".. some binary .." is actually binary data, and string is UTF8.

How does the chunked messages for CALL_RESULTs look like?

For example 4., is it

  [CALL_RESULT, [1, 2, 3] ]

plus something like this

  1 | 3 | "bar"
  2 | 10 | ".. some binary .."
  3 | 1 | "1"

But where is the type info for the third element left?

Anyway, WebSocket as a transport supports both text and binary messages, interleaved, but well ordered. A while ago I was contemplating about splitting out binary elements from payloads into binary WebSocket messages immediately following a text (JSON) based plain message that only contains pointers to the 2nd binary element containing WebSocket message. In a sense, somewhat similar to your proposal. But how do you mark up / differentiate a pointer to a binary element within the payload vs a plain value element?

from wamp-proto.

I was assuming type info would be known by the client, but maybe this isn't a valid assumption.

Perhaps something like this then:

[CALL_RESULT, [ STRING | 3 | "bar", BLOB | 1, INTEGER | 1 | "1" ] ]
1 | 10 | ".. some binary .."

Where each element has it's type, and either a size and a value (if not blob), or a pointer number. The standard JSON data-types would be defined: UTF-8 string, integer, floating-point number, array, object hash, boolean, null (covers undefined).

That's probably what you were thinking anyway.

I do think chunked encoding is important though, because this would allow the host to stream multiple large binary chunks at the same time. So if I requested two large files, I could do:

[ CALL_RESULT, [ BLOB | 1, BLOB | 2] ]
1 | 1024 | ".. some binary .."
2 | 1024 | ".. some binary .."
1 | 1024 | ".. some binary .."
2 | 1024 | ".. some binary .."
1 | 1024 | ".. some binary .."
2 | 1024 | ".. some binary .."
2 | 1024 | ".. some binary .."

Each chunk is written as soon as it is available. There will have to be a little buffering per WebSocket frame, but overall it would be more efficient than doing each file in series in terms of being able to pull data. For example, if I'm making a game, I can have audio, textures and models for the next scene loading at the same time and the audio can be played while everything is downloading.

It might make sense to store the total size of the BLOB, but there should be a way to stream without knowing the size, in case it's being generated on the fly.

from wamp-proto.

Having the ability to transport binary would be very useful, and makes sense since the underlying websocket protocol already supports it. It looks like you two pretty much have it figured out. Is there anything else that needs to be figured out before it could be implemented?

from wamp-proto.

WAMPv2 specifies two serializations: JSON and MsgPack (v5). The latter now includes distinct string and binary types.

WAMPv2 also addresses more of above discussion by make a clear distinction of what seriazation format features are used by WAMPv2 itself, and what is only relevant to application payload.

from wamp-proto.

Even though msgpack seems to support binary, the unpacker inside AutobahnPython doesn't seem terribly happy with it.

The data provider can pack the data just fine, but the consumer has trouble unpacking it as binary seems to be considered a utf8 string

I'll get this message :
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

from wamp-proto.

@mtourne - Issues specific to AutobahnPython are better reported at the AutobahnPython repository.

from wamp-proto.

Sure, I will do that too.
I was thinking the protocol itself could make provisions for binary data (and streaming binary data) independent of which serializer (or how the serializer) is used.

from wamp-proto.