Code Monkey home page Code Monkey logo

engine.io-protocol's Introduction

Engine.IO Protocol

This document describes the 4th version of the Engine.IO protocol.

Table of content

Introduction

The Engine.IO protocol enables full-duplex and low-overhead communication between a client and a server.

It is based on the WebSocket protocol and uses HTTP long-polling as fallback if the WebSocket connection can't be established.

The reference implementation is written in TypeScript:

The Socket.IO protocol is built on top of these foundations, bringing additional features over the communication channel provided by the Engine.IO protocol.

Transports

The connection between an Engine.IO client and an Engine.IO server can be established with:

HTTP long-polling

The HTTP long-polling transport (also simply referred as "polling") consists of successive HTTP requests:

  • long-running GET requests, for receiving data from the server
  • short-running POST requests, for sending data to the server

Request path

The path of the HTTP requests is /engine.io/ by default.

It might be updated by libraries built on top of the protocol (for example, the Socket.IO protocol uses /socket.io/).

Query parameters

The following query parameters are used:

Name Value Description
EIO 4 Mandatory, the version of the protocol.
transport polling Mandatory, the name of the transport.
sid <sid> Mandatory once the session is established, the session identifier.

If a mandatory query parameter is missing, then the server MUST respond with an HTTP 400 error status.

Headers

When sending binary data, the sender (client or server) MUST include a Content-Type: application/octet-stream header.

Without an explicit Content-Type header, the receiver SHOULD infer that the data is plaintext.

Reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type

Sending and receiving data

Sending data

To send some packets, a client MUST create an HTTP POST request with the packets encoded in the request body:

CLIENT                                                 SERVER

  │                                                      │
  │   POST /engine.io/?EIO=4&transport=polling&sid=...   │
  │ ───────────────────────────────────────────────────► │
  │ ◄──────────────────────────────────────────────────┘ │
  │                        HTTP 200                      │
  │                                                      │

The server MUST return an HTTP 400 response if the session ID (from the sid query parameter) is not known.

To indicate success, the server MUST return an HTTP 200 response, with the string ok in the response body.

To ensure packet ordering, a client MUST NOT have more than one active POST request. Should it happen, the server MUST return an HTTP 400 error status and close the session.

Receiving data

To receive some packets, a client MUST create an HTTP GET request:

CLIENT                                                SERVER

  │   GET /engine.io/?EIO=4&transport=polling&sid=...   │
  │ ──────────────────────────────────────────────────► │
  │                                                   . │
  │                                                   . │
  │                                                   . │
  │                                                   . │
  │ ◄─────────────────────────────────────────────────┘ │
  │                       HTTP 200                      │

The server MUST return an HTTP 400 response if the session ID (from the sid query parameter) is not known.

The server MAY not respond right away if there are no packets buffered for the given session. Once there are some packets to be sent, the server SHOULD encode them (see Packet encoding) and send them in the response body of the HTTP request.

To ensure packet ordering, a client MUST NOT have more than one active GET request. Should it happen, the server MUST return an HTTP 400 error status and close the session.

WebSocket

The WebSocket transport consists of a WebSocket connection, which provides a bidirectional and low-latency communication channel between the server and the client.

The following query parameters are used:

Name Value Description
EIO 4 Mandatory, the version of the protocol.
transport websocket Mandatory, the name of the transport.
sid <sid> Optional, depending on whether it's an upgrade from HTTP long-polling or not.

If a mandatory query parameter is missing, then the server MUST close the WebSocket connection.

Each packet (read or write) is sent its own WebSocket frame.

A client MUST NOT open more than one WebSocket connection per session. Should it happen, the server MUST close the WebSocket connection.

Protocol

An Engine.IO packet consists of:

  • a packet type
  • an optional packet payload

Here is the list of available packet types:

Type ID Usage
open 0 Used during the handshake.
close 1 Used to indicate that a transport can be closed.
ping 2 Used in the heartbeat mechanism.
pong 3 Used in the heartbeat mechanism.
message 4 Used to send a payload to the other side.
upgrade 5 Used during the upgrade process.
noop 6 Used during the upgrade process.

Handshake

To establish a connection, the client MUST send an HTTP GET request to the server:

  • HTTP long-polling first (by default)
CLIENT                                                    SERVER

  │                                                          │
  │        GET /engine.io/?EIO=4&transport=polling           │
  │ ───────────────────────────────────────────────────────► │
  │ ◄──────────────────────────────────────────────────────┘ │
  │                        HTTP 200                          │
  │                                                          │
  • WebSocket-only session
CLIENT                                                    SERVER

  │                                                          │
  │        GET /engine.io/?EIO=4&transport=websocket         │
  │ ───────────────────────────────────────────────────────► │
  │ ◄──────────────────────────────────────────────────────┘ │
  │                        HTTP 101                          │
  │                                                          │

If the server accepts the connection, then it MUST respond with an open packet with the following JSON-encoded payload:

Key Type Description
sid string The session ID.
upgrades string[] The list of available transport upgrades.
pingInterval number The ping interval, used in the heartbeat mechanism (in milliseconds).
pingTimeout number The ping timeout, used in the heartbeat mechanism (in milliseconds).
maxPayload number The maximum number of bytes per chunk, used by the client to aggregate packets into payloads.

Example:

{
  "sid": "lv_VI97HAXpY6yYWAAAC",
  "upgrades": ["websocket"],
  "pingInterval": 25000,
  "pingTimeout": 20000,
  "maxPayload": 1000000
}

The client MUST send the sid value in the query parameters of all subsequent requests.

Heartbeat

Once the handshake is completed, a heartbeat mechanism is started to check the liveness of the connection:

CLIENT                                                 SERVER

  │                   *** Handshake ***                  │
  │                                                      │
  │  ◄─────────────────────────────────────────────────  │
  │                           2                          │  (ping packet)
  │  ─────────────────────────────────────────────────►  │
  │                           3                          │  (pong packet)

At a given interval (the pingInterval value sent in the handshake) the server sends a ping packet and the client has a few seconds (the pingTimeout value) to send a pong packet back.

If the server does not receive a pong packet back, then it SHOULD consider that the connection is closed.

Conversely, if the client does not receive a pong packet within pingInterval + pingTimeout, then it SHOULD consider that the connection is closed.

Upgrade

By default, the client SHOULD create an HTTP long-polling connection, and then upgrade to better transports if available.

To upgrade to WebSocket, the client MUST:

  • pause the HTTP long-polling transport (no more HTTP request gets sent), to ensure that no packet gets lost
  • open a WebSocket connection with the same session ID
  • send a ping packet with the string probe in the payload

The server MUST:

  • send a noop packet to any pending GET request (if applicable) to cleanly close HTTP long-polling transport
  • respond with a pong packet with the string probe in the payload

Finally, the client MUST send a upgrade packet to complete the upgrade:

CLIENT                                                 SERVER

  │                                                      │
  │   GET /engine.io/?EIO=4&transport=websocket&sid=...  │
  │ ───────────────────────────────────────────────────► │
  │  ◄─────────────────────────────────────────────────┘ │
  │            HTTP 101 (WebSocket handshake)            │
  │                                                      │
  │            -----  WebSocket frames -----             │
  │  ─────────────────────────────────────────────────►  │
  │                         2probe                       │ (ping packet)
  │  ◄─────────────────────────────────────────────────  │
  │                         3probe                       │ (pong packet)
  │  ─────────────────────────────────────────────────►  │
  │                         5                            │ (upgrade packet)
  │                                                      │

Message

Once the handshake is completed, the client and the server can exchange data by including it in a message packet.

Packet encoding

The serialization of an Engine.IO packet depends on the type of the payload (plaintext or binary) and on the transport.

HTTP long-polling

Due to the nature of the HTTP long-polling transport, multiple packets might be concatenated in a single payload in order to increase throughput.

Format:

<packet type>[<data>]<separator><packet type>[<data>]<separator><packet type>[<data>][...]

Example:

4hello\x1e2\x1e4world

with:

4      => message packet type
hello  => message payload
\x1e   => separator
2      => ping packet type
\x1e   => separator
4      => message packet type
world  => message payload

The packets are separated by the record separator character: \x1e

Binary payloads MUST be base64-encoded and prefixed with a b character:

Example:

4hello\x1ebAQIDBA==

with:

4         => message packet type
hello     => message payload
\x1e      => separator
b         => binary prefix
AQIDBA==  => buffer <01 02 03 04> encoded as base64

The client SHOULD use the maxPayload value sent during the handshake to decide how many packets should be concatenated.

WebSocket

Each Engine.IO packet is sent in its own WebSocket frame.

Format:

<packet type>[<data>]

Example:

4hello

with:

4      => message packet type
hello  => message payload (UTF-8 encoded)

Binary payloads are sent as is, without modification.

History

From v2 to v3

  • add support for binary data

The 2nd version of the protocol is used in Socket.IO v0.9 and below.

The 3rd version of the protocol is used in Socket.IO v1 and v2.

From v3 to v4

  • reverse ping/pong mechanism

The ping packets are now sent by the server, because the timers set in the browsers are not reliable enough. We suspect that a lot of timeout problems came from timers being delayed on the client-side.

  • always use base64 when encoding a payload with binary data

This change allows to treat all payloads (with or without binary) the same way, without having to take in account whether the client or the current transport supports binary data or not.

Please note that this only applies to HTTP long-polling. Binary data is sent in WebSocket frames with no additional transformation.

  • use a record separator (\x1e) instead of counting of characters

Counting characters prevented (or at least makes harder) to implement the protocol in other languages, which may not use the UTF-16 encoding.

For example, was encoded to 2:4€, though Buffer.byteLength('€') === 3.

Note: this assumes the record separator is not used in the data.

The 4th version (current) is included in Socket.IO v3 and above.

Test suite

The test suite in the test-suite/ directory lets you check the compliance of a server implementation.

Usage:

  • in Node.js: npm ci && npm test
  • in a browser: simply open the index.html file in your browser

For reference, here is expected configuration for the JavaScript server to pass all tests:

import { listen } from "engine.io";

const server = listen(3000, {
  pingInterval: 300,
  pingTimeout: 200,
  maxPayload: 1e6,
  cors: {
    origin: "*"
  }
});

server.on("connection", socket => {
  socket.on("data", (...args) => {
    socket.send(...args);
  });
});

engine.io-protocol's People

Contributors

1c3t3a avatar albertyfwu avatar darrachequesne avatar defunctzombie avatar haykokoryun avatar miguelgrinberg avatar minchao avatar pkyeck avatar rauchg avatar sweetiesong avatar totodore avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

engine.io-protocol's Issues

Binary messages should be of a different packet type to plaintext messages.

Due to the nature of data being transmitted, message packets with a binary payload need different handling (identifying, encoding, decoding). This is already being done, but the specification attempts to make these types agree by attempting to bunch them together under a single packet message with ID 4, but the same specification, on the other hand, also identifies binary packets differently (using a b character).

So, given that not only are the packets handled differently, stored differently, but also is identified differently and transmitted differently (binary packets have MIME type application/octet-stream, text packets have MIME type text/plain), wouldn't it simply make more sense to have two message packet types, rather than trying hard to make them coexist under a single packet type as is currently being done?

Isn't it a `ping` packet that the client waiting for?

Conversely, if the client does not receive a pong packet within pingInterval + pingTimeout, then it SHOULD consider that the connection is closed.

The Hearbeat section says, server send ping to client, then the client reply a pong to server. So the client should wait for a ping packet, is it a typo or?

Wrong about the ping description

In fact, the pong message doesn't contain the message send in ping

Where we can find the pong implement in engine.io:

switch (packet.type) {
      case 'ping':
        debug('got ping');
        this.sendPacket('pong');
        this.emit('heartbeat');
        break;
}

As we can see, it just send the pong message and ignore the ping payload.

Should update document like below:

- Sent by the client. Server should answer with a pong packet containing the same data
+ Sent by the client. Server should answer with a pong packet

client sends: 2probe
- server sends: 3probe
+ server sends: 3

Or change the implement of engine.io to fix this issues?

Lack of Version History

There should be a branch for each version of the protocol, as there are branches on the engine.io repository for 3.1.x, 2.1.x, etc. I have found it extremely frustrating to develop a client implementation without this necessary information.

Does packet type have two groups values in binary?

I'm the author of go-socket.io. I try to catch up current implement now. And I have some questions need to clarify.

As protocol described not very clear, if a packet is string, the leading type id is "4" in string. And if a packet is binary, the leading type id is a byte 4. Am I right?

So it can use type id to check if a packet is string or binary. Right?

And from implement, it seems only message(type 4) packet could be binary packet. All other types packet are string. So it only have types below:

  • "0": open
  • "1": close
  • "2": ping
  • "3": pong
  • "4": string message
  • byte 4: binary message
  • "5": upgrade
  • "6": noop

Is it right?

Is 'sid' parameter designed for reconnecting?

sid: if the client has been given a session id, it must be included in the querystring.

I'm connecting server without this param and it works well, so I'm wondering is this parameter just designed for reconnecting?

Thanks.

Payload encoding is unclear

The description of how payloads are encoded is unclear. In particular, how lengths should be encoded is not clear, there is conflicting information, and information that doesn't make sense.

Here are some unanswered questions that the spec leaves begging:

  • What is the length when the payload is binary encoded (ie, when XHR2 is supported)? Is it the number of bytes? Is it the number of bytes + 1? Previously, for non XHR2 requests, the length is defined as the number of characters, does that apply to XHR2 requests, or for XHR2 requests is it always the number of bytes?
  • For non XHR2 request, should be "b" for binary packets be included in the length or not? If so, then when it says that the length should be the length of the base64 encoded data + 1, that's wrong, because it's +1 for UTF8 packets, but +2 for binary packets.

Question about which side sends `upgrade` message.

In the front, I'm not an English native speaker. It may be my misunderstanding. I just want to make it clear.

In Sample session, WebSocket frames show

...
> 5         => "upgrade" packet type
...
> 2         => "ping" packet type

As ping is always sent by server-side and they share the same direction, I read it as upgrade is sent by server-side too. (That direction is different from what it is in the next session, but it doesn't relate with this question.)

In Transport Upgrading, it says

A connection always starts with polling (either XHR or JSONP). WebSocket gets tested on the (client) side by sending a probe. If the probe is responded from the server, an upgrade packet is sent (from which side? Client?).

As it says When the server receives the upgrade packet in the following paragraphs, it conflicts with WebSocket frames.

Could you clarify that the upgrade is sent from client and update the WebSocket frames?

Make decoder callback based

It should emit a callback with each packet:

decodePayload(packetsString, function(packet, isLast){});

Callback arguments:

  • packet just like each item of the returned array right now
  • isLast true if it's the last packet in the payload, false otherwise.

Instead of returning an array like it is now.
This will give us the possibility of giving events to the user before all packets are parsed from the string (important for very big payloads).

Let's make sure to add tests for this.

Usage XHR vs JSONP docs?

I can't find information on when JSONP is used and when XHR is used.
I checked the readme's from:

  • this repository
  • engine.io
  • engine.io-client

I think I understand one of these is used when a client is in "polling" mode, but there is little information on why one of these would be selected.
I understand I can force JSONP using the forceJSONP option and I could disable it by setting jsonp to false.

Question about long polling and duplicate poll requests

In test-suite.js there's a test that checks if the server closes the session and sends a 1 close packet when the client makes duplicate poll requests at the same time here, I have two questions about that:

  1. Why is it a problem that the client can make duplicate poll requests?
  2. What is the best way for a server to detect that the client made a duplicate poll request i was thinking of doing something with a timestamp we can attach a timestamp to the session and check it when the client makes a poll request.

I'm making my own engine.io implementation and I never done something like this before so i apologize for my ignorance, thanks :).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.