n0-computer / beetle Goto Github PK

Dir etags are not utilized and need to fully resolve each time.

Since the watch command, the gateway (and presumably other services) are unresponsive. On gateway it's due to https://github.com/n0-computer/iroh/blob/main/iroh-gateway/src/core.rs#L87, the call never returns as it's basically starting the tonic server on the current thread and essentially waiting for it to halt. Should probably spin it off similar to https://github.com/n0-computer/iroh/blob/main/iroh-gateway/src/main.rs#L49

I'm looking into leveraging iroh-ctl as a hub to report on some stats, for that it would be ideal if I could list and tag different RpcClientConfig so that it watches over all of them. Ideally I could do aggregate result vs per tag/namespace/somekey.

ie iroh-ctl status --watch --cfg=./mycfg.[toml|yaml|json] & iroh-ctl status --watch --tag=node_omega --cfg=./mycfg.[toml|yaml|json]

If we had this, I'd be happy to add an option to run --with-metrics so that it reports some sort of uptime metric. Useful for a central health/uptime monitoring solution.

• multiple providers (all have the content)
  • make sure cancels are sent out
• multiple providers (some are not dialable)
  • make sure content is fetched & failed dials are cleaned up
• no dialable providers
  • make sure the request times out
• benchmarks
  • 1-1 transfer on the same machine, (msg sizes: 128, 256, 4KiB, 16KiB, 512KiB 1MiB, 2MiB)

The current windows actions environment runs out of disk space.
Either use hacks to remove stuff from the environment on each run or self host runners.

Should support both v1 and v2

Ref: https://ipld.io/specs/transport/car/

Serve pretty 404 where possible.

Improve iroh-store metrics & add underlying rocksdb metrics.

We're starting to see OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full which is usually the result of performance limitations on the collector side.

The gateway must be deployed to the cloud:

• need AWS accoutn attached to a n0 credit card
• need a strategy for deploying, ideally hooked into CI

Metrics dashboards need to be updated with the latest knowledge and metrics.
Ops game needs to be improved - propper auth, backups, configuring storage, retention policies, rollups etc.
Add new dashboards: libp2p, dht, storage

• gateway needs updated config & depends on storage/p2p
• p2p needs to be added to the list of services and configured
• storage needs to be added to the list of services and configured

Dir listing is very plain. We should list size where possible, breadcrumbs to parent, potentially resolve paths to list items.
Improve overall template styling, add execution time.

Record detailed metrics about bitswap.

Some inspiration can be taken from https://github.com/ipfs-rust/libp2p-bitswap/blob/master/src/stats.rs

Must be able to launch the gateway, network, & storage processes via the command line.

Should be able to add configuration that allows each of the three processes to know about each other, using a config that specifies what address that process should be listening on, and the addresses of the processes to which it should connect.

Look into testground
Setup 1st integration test

Would be nice if iroh-ctl had a readme

• spawn iroh-p2p
• spawniroh-gateway
• request http://localhost:9050/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR from the gateway
• assert it is the correct image data

• Client Mode: libp2p/rust-libp2p#2521
• Accelerated DHT
  • https://github.com/libp2p/go-libp2p-kad-dht/blob/master/fullrt/dht.go
  • https://www.notion.so/pl-strflt/DHT-Overview-e4bc573eaf094efc8a0781de47262db6#1f7fe618623a473dac5580a0f9609616
• Make timeouts configurable
• Return found providers before the full query has returned
• Make the number of requested providers configurable

traces use a TON of storage and bw, we should have those configurable and able to compile without traces.

Streaming content from the gateway is set up for the happy path. Breaking the connection or anything along the lines just errors out. Ideally we should track those requests, add metrics & handle errors better.

depends on #280

Gateway is half baked right now and needs some love on putting the resolver in the right place, adding metrics and cleaning up the client.

iroh-ctl status --watch

iroh-ctl status

We already have iroh-car to read car files, so now this needs to be hooked up through the cli so we can ingest large files of data into the store.

See https://docs.rs/libp2p/latest/libp2p/tutorials/hole_punching/index.html for instructions on how to set this up

Should add gossipsub, and rpc commands & cli commands ideally to interact with it

Ref: https://docs.rs/libp2p/latest/libp2p/gossipsub/struct.Gossipsub.html

maybe store the Path directly instead of the stringified version?

Originally posted by @dignifiedquire in n0-computer/iroh#87 (comment)

Very little care is given for properly handling or responding to utf-8 based urls or headers.

Most services will need some configuration support. Should get started on having a centralized ~/.iroh config folder.
dig suggests taking a look at https://github.com/mehcode/config-rs/blob/master/examples/hierarchical-env/settings.rs

If a config file exists, pull from it. Otherwise use defaults or env variables.

At this stage, if you want to use a config file, you must manually add it yourself to the iroh "home" folder /.iroh.

As a first round, we are only focusing on the ability to add custom rpc addresses.

Currently error handling is pretty scattered and we have lots of unchecked unwraps etc.

We've got the basic building blocks in place however many pieces are either not instrumented, don't have counters or lose the trace context between internal over the wire calls.

Currently many requests go and invoke fetch_bitswap with an empty provider list (ref trace ID 2b90b66cb34b936f005cde120d35de0e). We should fail fast here and return 404 or something similar.

The response from the resolver on gateway should be a stream, both to avoid loading everything into memory and significantly reduce the time to first byte which is currently t(fetch_bitswap) + some_overhead while it could be t(fetch_bitswap_first_byte) + some_overhead. This also allows us to return metrics like time to first byte/block, latency vs time etc.

We should have a set of peers (configurable size) that we store in memory, as main peers. These can be used to send bitswap requests to, as well as improve general connectivity.

Ops game needs to be improved - propper auth, backups, configuring storage, retention policies, rollups etc.

https://github.com/RustSec/rustsec/tree/main/cargo-audit

We're flying blind, we should have alerts for basic uptime issues.

HEAD requests are not accepted and not set up to properly respond even if we allow it.

Create RPC client & network server implementations/

The goal is to have something like this run:

    async fn main() {
        let client = iroh_rpc_client::new().await.unwrap();
        let c = client.close();
        let task = tokio::spawn(|| async move { c.run().await; });

        let my_bytes = client.network.get("foobar").await.unwrap();
        println!("mybytes: {:?}", my_bytes);

        task.join().await.unwrap();
    }

Crates:

• iroh-rpc
• iroh-rpc-client
• iroh-rpc-network

pub mod rpc {
    //! equivalenet to grpc or jsonrpc_v2
    //!
    //! - manages libp2p node
    //! - manages addresses
    //! - manages serialization
    //! - manages all internal comms

    pub struct Client {
        server: Libp2p,
        config: ClientConfig,
    }

    pub struct Server {
        server: Libp2p,
        config: ServerConfig,
        client_config: ClientConfig,
        state: State<T>
        router: R
    }
}

pub mod iroh_rpc_client {
    pub fn client_config() -> rpc::ClientConfig {
        rpc::ClientConfig::new()
          .with_namespace("db", |db| {
              db.with_method("get", crate::db::get)
          })
          .with_namespace("network", |n| {
              n.with_method("get", crate::network::get)
          })
    }

    pub fn new() -> Client {
        rpc::Client::with_config(client_config())
    }

    mod db {
        pub async fn get(cid: Cid) -> Result<()> {
            make_stream_request(DbRequest::Get(cid))
        }
    }

    mod network {
        pub async fn get(cid: Cid) -> Result<()> {
            make_stream_request(NetworkRequest::Get(cid))
        }
    }
}


pub mod iroh_network {
    // /network/get
    fn new_rpc_server(server_state: NetworkState) -> rpc::Server {
        let server = rpc::Server::new()
          .with_state(server_state)
          .with_namespace("network", |n| {
              n.with_method("get", network::get)
          })
          .with_client_config(iroh_rpc_client::client_config());
        server
    }

    mod network {
        pub async fn get(state &NetworkState, cid: Cid) -> Result<Bytes> {
            let result = state.get_cid(cid).await;
            result
        }
    }
}

pub mod iroh_db {
    // /db/get
    fn new_rpc_server(db_state: DbState) -> rpc::Server {
        let server = rpc::Server::new()
          .with_state(server_state)
          .with_namespace("db", |n| {
              n.with_method("get", db::get)
          })
          .with_client_config(iroh_rpc_client::client_config());

        // we can now make requests: server.client().namespace("db").call("get", "foobar").await;
        server
    }

    mod db {
        pub async fn get(state &DbState, cid: Cid) -> Result<Bytes> {
            let result = db.get(cid).await;
            result
        }
    }}

pub mod iroh_cli {
    async fn main() {
        let client = iroh_rpc_client::new().await.unwrap();
        let c = client.close();
        let task = tokio::spawn(|| async move { c.run().await; });

        let my_bytes = client.db.get("foobar").await.unwrap();
        println!("mybytes: {:?}", my_bytes);

        task.join().await.unwrap();
    }
}

Based on #276
Passing trace contexts over our internal RPC messaging bus still isn't implemented.

We should start working on a load balancer for the gateway.
It's still early to have real use for it but it's much easier to evolve and grow with us.
Reasons to do it:

• we already have 2 gateways up, I'd love to have seamless handoff between the two
• puts us up on the stall to work on the "real stuff" from day 1
• should be easy enough to get started

Why not use something off the shelf

• I can se us getting into the weeds with this: I'd love to work on supporting % rollouts, feature flagging on the gateway level, failover mechanics, geo based balancing, rtt based balancing, contention based etc... Don't think anything of the self will give us enough flexibility

Fail/Error count are only recorded if we serve a gateway reponse/error directly. Error handling swallows the response and never triggers the counters.

Also trace ID is missing from such cases as the trace ID is resolved from within the call and not globally.

Our header logic for cache control is very crude. Either we say to the browser to cache forever if it's a ipfs/CID or simply set the Last-Modified header to now().

We should explore cache logic around ipns/dns TTLs, content publishing timestamp for Last-Modified

• the store binary needs to be hooked up the metrics
• the store itself should record metrics about its usage, including stats from rocksb

• Add types/methods for in iroh-rpc-types:
  put

pub async fn put<T: AsRef<[u8]>, L>(&self, cid: Cid, blob: T, links: L) -> Result<()>
    where
        L: IntoIterator<Item = Cid>

get

pub async fn get(&self, cid: &Cid) -> Result<Option<DBPinnableSlice<'_>>>

get_links

pub async fn get_links(&self, cid: &Cid) -> Result<Option<Vec<Cid>>>

• add store.rs client & associated methods
• add iroh-rpc-store::rpc mod that can create a store specific server & knows how to respond to rpc requests
• add rpc_addr to the store Config
• run StoreServer rpc in mainso we can communicate with the store.
• add missing p2p rpc commands