Browser support for Irmin-server

Problem

There is need to support communication between the browser and irmin-server.
Currently irmin-server uses flows to communicate between the server and the client. Flows are byte-streams like a TCP connection and these are established using Conduit. In the browser, there is currently no mechanism we know of for having a bi-directional, stream-oriented communication channel

The closest thing is a websocket which solves bidirectional communication, but is inherently message-oriented and not stream-oriented. Websockets sound like a good solution but it could be implemented in a few different ways.

Solutions

A conduit-like flow using websockets

One solution is to impose a stream-like structure layered on top of the message-oriented websocket connection. This could look like the following for example:

let websocket_handler server client =
    let flow = Obj.magic () in (* BAD! *)
    let rec fill_ic channel client =
      let* frame = Websocket_lwt_unix.Connected_client.recv client in
      Lwt_io.write channel frame.content >>= fun () ->
      fill_ic channel client
    in
    let rec send_oc channel client =
      Lwt_io.read channel >>= fun content ->
      Websocket_lwt_unix.Connected_client.send client (Websocket.Frame.create ~content ()) >>= fun () ->
      send_oc channel client
    in
    let input_ic, input_oc = Lwt_io.pipe () in
    let output_ic, output_oc = Lwt_io.pipe () in
    Lwt.async (fun () -> fill_ic input_oc client);
    Lwt.async (fun () -> send_oc output_ic client);
    callback server flow input_ic output_oc

Although it is unclear if this will be sufficient but the basic idea is to have two pipes for reading and writing. When we receive a websocket message we add it to the reading channel and when we send a message we add it to the writing channel.

Reimplement protocol but with message-oriented

Going this route means that most features has to be reimplemented to support a message-oriented communication
1. The IO module - The IO module currently uses Conduit_lwt_unix, which communicates using flow.
2. The protocol - The protocol is currently structured around conn_intf which calls the IO module which uses flow.

Questions

1. What would the API look like if we support a message-oriented and stream-oriented communication.

This could be a result of re-using the types from Store.Backend

When over 30,000 commits are replayed it looks like the server is being killed because it's running out of memory. A smarter way of managing trees on the server should fix this.

It would be nice to release this after Irmin 3.0 is released.

TODO

• Ensure compatibility with Irmin
• Implement Irmin.S using Irmin_client (#34)
• Fix irmin slow tests
• Add more tests (Irmin_client.Store, replicate command, ...)

I have this partially working and it reveals that websockets from the browser don't pass the tests, I'll try to fix the bugs and start upstreaming all of the various changes needed to put irmin-test in the browser.

Currently there are two different APIs that require two different sets of commands to implement. It seems like a good idea to focus on the Store client API with some additional functions (ping, reconnect, ...)

This would require:

• Removing a lot of the existing commands not found in command_backend.ml
• Moving some functionality from Irmin_client.Make to Irmin_client.Store.Make
• Removing unused functionality from irmin-server-internal
• Porting Batch API to Store client

It should be possible to use Blob or Uint8Array on the Javascript side to support binary websockets, which would make it possible to remove the base64 encoding that websocket payloads go through.

Will there be a version of irmin-client without the lwt?