Goal

Start a new Beaker Browser 1.0 share on the computer, and find shared folder over mDNS message broadcast.

What we gonna do on the live stream?

This goal can be divided in two sub-goals: One to "Find Hyperdrives over LAN" and another to "Announce a Hyperdrive over LAN". This is similar to colmeia-dat1-mdns, but with a new protocol and data exchange.

We'll start with locating a Hyperdrive, and if we have time, announce our own hyperdrive as well.

Locate a Hyperdrive share

• Find the packets on the network using Wireshark
• Create a new UDP Socket in Rust that listen and broadcast mDNS packets
• Query the network using mDNS packets
• Create two async process: One to query, one to listen
• Parse Beaker packets and print the information received
• Expose a stream of IPs collected
• Deal with errors instead of crashing with unwrap/expect's

Announce a Hyperdrive share

• Find the packets on the network using Wireshark
• Create a new UDP Socket in Rust that listen and broadcast mDNS packets
• Listen to Beaker mDNS queries and print them
• Create two async process: One to announce, one to respond
• Respond with a well formatted query and check if Beaker tries to connect
• Deal with errors instead of crashing with unwrap/expect's

References

• https://github.com/hyperswarm/discovery
• colmeia-dat1-mdns

Track: https://github.com/mattsse/hyperswarm-dht

• Find peers over DHT

This design seems to be able to run multi-threaded/tasks already given the colmeia-server implementation.

Is this a good design to implement new servers/clients?

Good:

• Allows composition
• Allows storing state and mutation (useful for tracking/hypercore)
• allows early connection close

Bad:

• Verbose to implement?
• Repetitive to implement
• Not the same design as the Node version

Nowadays, colmeia-sync and ffi panics when it tries to talk to itself, when the socket closes and the writer fails.

It would be nice to remove all unwraps and expect and change signatures to return errors as needed.

Ideally, ProtocolEvents should have a associated error type to allow use of other errors by the consumer of the trait.

Some places have self.stream_on_network.write().next().await which may lead to locking the stream for a long time.

Some places use .timeout from async-std to avoid locking it for too long, but the timeout is arbitrary. Instead, it would be better to use .now_or_never() from future crate.

This would make it easier to use also make the executor abstract on #25

Currently, colmeia has been working with as single hypercore concept.

@Frando has been very kind to share some of his early work on his hypercore-replicator crate, which the design supports replicating multiple feeds.

Hyperdrive, the filesystem abstraction built on top of hypercores, has to replicate at least 2 cores.
This is noticeable when debuging MDNS discovery messages where we receive multiple feeds at once.

This is also how JS is replicating the data. Now, there is a corestore and the hyperspace API in JS, which is this broader abstraction of maintaining multiple feeds in sync at once.

To align better with @Frando's hypercore-replicator, which is a quite nice API, this issue documents the need to change how Colmeia deals with single feeds vs multiple feeds.

Concern

Currently, to replicate and discover multiple hypercores in colmeia, it would need to create multiple Hypercore Stacks (as on #21), which means opening multiple sockets at once. That means that each hyperdrives (pairs of cores) would allocate one listener socket, one discovery mdns responder socket and one mdns discovery. Given we are targeting mobile (and possibly routers), that is a bit wasteful.

It would better fit to redesign colmeia-hyperswarm-mdns to be able to add new feed keys (and removal), so we can discover multiple feeds on the network with a single socket.

Currently, this libraries since #21 has been using async-std to spawn new tasks.

We could instead offer the task executor to be configurable using cargo feature selectors. There is the https://github.com/bastion-rs/agnostik crate to help with this.

This would let the code to be also used on Tokio or Bastions codebases (preferring using async-std or smol by default, given that we still make use of async-std::RwLock on other stuff.

I have no idea what a proactive IO framework is as well - but it would be worth checking if it make sense to use (https://github.com/vertexclique/nuclei)

Given that colmeia-hyperswarm-mdns crate is using my own custon multicast-socket crate, it would not benefit from async-std's epoll io or bastion-rs/nuclei. The replicator crate would be the next one that need to find a good io replicator.

TODO on a livestream:

• Include bastions-rs/agnostik crate on every crate that spawns tasks
• Add feature selector to select which crate to use

Investigation:

• https://github.com/bastion-rs/agnostik
• https://github.com/vertexclique/nuclei
• https://crates.io/crates/multitask
• https://crates.io/crates/async-task
• An alternative to async-std::sync::RwLock so we can leave it as an optional dependency - https://crates.io/crates/async-rwlock https://crates.io/crates/futures-locks

I found that running the discovery on mdns only worked for one interface. As soon as I tried it on a multihomed device, such as my laptop with WiFi and Cable on different networks, I could not find it anymore.

The discovery has been documented on socket-2, which shows that while we can listen on multiple interfaces for the incoming packets, we need to do some gymnastics to send packets on multiple interfaces.

Goal

Create a single socket which listen to multicast packets on multiple interfaces, and can send packets on a multiple interfaces as well.

What we gonna do on the live stream?

The goal is to create a socket interacting with socket-2 that provides an API to multiple interfaces. This will require having Rust interact with C++/C structs and do C->Rust FFI (which I have no idea how to do that much.) Come learn with me.

Create a Multicast UDP socket that:

• Listens for a specific multicast ip:port (eg: MDNS)
• Join multicast group on multiple interfaces
• Receive a message: (data, origin ip, origin interface)
• Send a message: (data, destination ip, origin interface)
• Send a message on all active interfaces: (data, destination ip, [all interfaces])

References

• rust-lang/socket2#86
• https://stackoverflow.com/questions/32278414/c-listen-to-multicast-on-all-interfaces-respond-on-same-as-recieved
• https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms741687(v=vs.85)
• rust-lang/socket2#29
• nix-rust/nix#990

Dat Protocol is moving into a new version, that is not backward-compatible.

To help to communicate this breaking change Dat announced it is moving to Hypercore Protocol.

Dat (also known as dat1 on this repo) will keep existing as is, but the same team that maintained the protocol (also known as dat2 on this repo) is now focused on adding features only on the new protocol.

Also, the new Beaker only works on the new protocol, and will migrate the library on previous Dat to Hypercore.

Apps that were targeting Dat will not work with the new protocol, as discovery also changed.

This is tracking issue to move colmeia to the Hypercore Protocol:

• Discovery: mdns hyperswarm discovery #11
  • Fix multi-interface discovery #15
• Handshake & Exchange with hypercore protocol over network #18
• Update hypercore version used on the code to the beta releases #21
• Change https://github.com/bltavares/parse-dat-url to parse-hyper-url #21
  • Removed as hypercore does not have dns resolution anymore beakerbrowser/beaker#1576
• Move dat bins to hypercore protocol #21
  • All dat bins moved into legacy for historical purpose
  • All previous bins available as hypercore
• Move dat1 from the repo to avoid dependencies breaking the latest code

Now that we have multicast, I would like to be able to interact with the beaker we found on the network.
We can test this by creating a new binary similar to colmeia-nc but using hypercore instead of dat.

So we will:

• Import https://github.com/Frando/hypercore-protocol-rs
• Handshake with each other
• Print the handshake and finalization on the terminal

We are already on a point where we could try to integrate with hypercore.

We are able to connect to a npx dat share running on another computer, both using the colmeia-nc or the colmeia-server.

https://gist.github.com/bltavares/ec688b8e6664834c51fe373f8c07c314

I've noticed that npx dat sends us two feed items for a simple dat repo, and from now on, this seems to be part of the replication stream. Maybe a dat repo has many hypercore feeds?

Hey! Nice project by the way 🙂

So I've being trying to compile it and having trouble on it.

I've tried just cloning and using cargo build but I get this output:

╭─otaviopace at mettaton in /Users/otaviopace/colmeia (master ✔)
╰─λ cargo build                                                                                                                cross-0 | 101 < 02:11:17
    Updating git repository `https://github.com/async-std/async-std`
error: failed to resolve patches for `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to load source for dependency `async-std`

Caused by:
  Unable to update https://github.com/async-std/async-std?branch=fix-atomic-32#64844deb

Caused by:
  revspec '64844debcb15dc8e7fd2340577da24dbcaeb781a' not found; class=Reference (4); code=NotFound (-3)

After reading some of your README.md, I've seen that you use cross to compile for Android. So I've tried using it too to compile, and I had the same issue 🤔

╭─otaviopace at mettaton in /Users/otaviopace/colmeia (master ✔)
╰─λ cross build --target armv7-linux-androideabi                                                                                  fish-0 | 0 < 02:10:27
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2020-06-04, rust version 1.44.0 (49cae5576 2020-06-01)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 60.2 MiB /  60.2 MiB (100 %)  50.6 MiB/s in  1s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 12.2 MiB /  12.2 MiB (100 %)   8.6 MiB/s in  1s ETA:  0s
info: installing component 'rust-std'
info: installing component 'rustc'
 60.2 MiB /  60.2 MiB (100 %)  15.0 MiB/s in  4s ETA:  0s
info: installing component 'rustfmt'

  stable-x86_64-unknown-linux-gnu installed - (error reading rustc version)

info: checking for self-updates
info: downloading component 'rust-std' for 'armv7-linux-androideabi'
info: installing component 'rust-std' for 'armv7-linux-androideabi'
Unable to find image 'rustembedded/cross:armv7-linux-androideabi-0.2.0' locally
armv7-linux-androideabi-0.2.0: Pulling from rustembedded/cross
fe703b657a32: Pull complete
f9df1fafd224: Pull complete
a645a4b887f9: Pull complete
57db7fe0b522: Pull complete
c3de58f51717: Pull complete
f8aac8546953: Pull complete
e0fe7a974cca: Pull complete
de899c6f7dca: Pull complete
abf70cb17661: Pull complete
f707b04b4965: Pull complete
3b162ba5b74b: Pull complete
67d0daeca688: Pull complete
c2de46407807: Pull complete
fa889352c751: Pull complete
c1779a84f287: Pull complete
72a6883f1f42: Pull complete
2d3e8798ffc5: Pull complete
Digest: sha256:853737710515f5de9a1442676c38065fc1f6dcd3f57904ee10f63814c70661a8
Status: Downloaded newer image for rustembedded/cross:armv7-linux-androideabi-0.2.0
    Updating git repository `https://github.com/async-std/async-std`
error: failed to resolve patches for `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to load source for dependency `async-std`

Caused by:
  Unable to update https://github.com/async-std/async-std?branch=fix-atomic-32#64844deb

Caused by:
  revspec '64844debcb15dc8e7fd2340577da24dbcaeb781a' not found; class=Reference (4); code=NotFound (-3)

So I have two questions:

• Do I have to do some kind of change on my Rust installation to async-std actually work on my computer?
• Does your project work compiling just for Windows/Mac/Linux, or everytime you do changes you have to use the cross compilation?

By the way I had the same issue on MacOS and Linux, but the logs are all on a Mac machine. Here is my installation info:

╭─otaviopace at mettaton in /Users/otaviopace/colmeia (master ✔)
╰─λ rustup update                                                                                                               fish-0 | 101 < 02:13:17
info: syncing channel updates for 'stable-i686-pc-windows-gnu'
info: syncing channel updates for 'stable-x86_64-apple-darwin'
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-apple-darwin'
info: checking for self-updates

       stable-i686-pc-windows-gnu unchanged - (rustc does not exist)
       stable-x86_64-apple-darwin unchanged - rustc 1.44.0 (49cae5576 2020-06-01)
    stable-x86_64-pc-windows-msvc unchanged - (rustc does not exist)
  stable-x86_64-unknown-linux-gnu unchanged - (error reading rustc version)
      nightly-x86_64-apple-darwin unchanged - rustc 1.46.0-nightly (118b50524 2020-06-06)

info: cleaning up downloads & tmp directories

What are the opportunities to improve the code?

I would like to start adding tests now that I'm able to connect to dat's and validate the design as we go.

There are lots of clone. Is there any chance to reduce memory allocation?

Instrument the code with https://crates.io/crates/tracing to have better visualization of how the system is behaving

The colmeia-hyperdrive crate is conflating the storage, peering, hyperdrive and hypercore code.

With @Frando's upcoming hypercore-replicator, we should be able to handle multiple replicated feeds, and we can focus on writing only the hyperdrive code, without conflating that much.

TODO Next livestream:

• Split colmeia-hypercore into colmeia-hyperdrive, and only use hyperdrive code.
• Adopt hypercore-replicator from @Frando on hyperstack
• Adopt updated hyperstack on colmeiad
• Bonus: Change protobuf crate to use the same as hypercore protocol (reduce dependencies) PROST depends on protoc