fedimint / fedimint Goto Github PK

Hi everyone,

thanks for your work on minimint. I saw there was already an issue of creating a WASM client library in #61 ... this would work for web apps but unfortunately there doesn't seem to be a way to load a .wasm module in react native directly. This is because The JS runtimes on iOS and Android don't support wasm yet.

There seems be some initial work to polyfill this with react-native-wasm. But that does not really look production ready. This is also why BlueWallet has built a React Native module for LDK called rn-ldk. This approach compiles the rust code to native iOS and Android libraries and uses Java/Swift bindings to call into native code from the JavaScript environment. I would propose to create a similar module for react native apps for the mint-client module. An example of how to do this can be found here.

Apart from having access to powerful native apis like QR code scanning on mobile, this way we could also leverage Photon SDK to build mobile apps.

Curious to hear your thoughts. Thanks :)

To avoid breaking #64 again we should have a test running on M1 in CI.

Currently consensus does not scale with available threads and generates empty epochs which wastes resources. In order to improve performance and simplify the consensus model we can adopt the following pub/sub model:

Create a proposal queue which allows multiple threads to submit ConsensusItem to be included in the next proposal.

• The API thread will submit new transactions to the queue
• Consensus threads can submit new items as a result of processing a ConsensusOutcome (e.g. peg-out sigs, blind sigs)
• Modules may also spawn threads to generate consensus items based on external events (such as the bitcoin block height increasing).
• The proposal queue will filter out all items from past proposals until a consensus thread removes the past proposal, indicating that the outcome of that proposal has been processed.

Any number of consensus threads which will consume consensus items to generate a ConsensusProposal, run the proposal through HBBFT, and process the outcome.

• Consensus threads are ordered and must generate proposals and process consensus outcomes in round-robin order (otherwise consensus state might become inconsistent between peers)
• Consensus threads generate a proposal if there are any consensus items or if they receive an HBBFT message indicating a proposal has been started by a peer.
• Proposals are guaranteed to be based on outcomes after n epochs, where n is the number of consensus threads (relevant for dropping non-contributing peers).

These changes also suggest that modules should move towards a simpler message-based pattern: receive new txs / outcome items and send consensus items. This would help decouple the modules from the consensus logic, eventually allowing them to even run remotely.

Currently hbbftand our tbs crate use different bls12-381 crates. Why is it that way, was bls12-381 split out of pairing at some point? If so can we upgrade hbbft to use the newer version? (we have a fork already, might as well keep our deps up to date)

We already use tracing for logging, but could be producing more structured and thus useful data that could be displayed in e.g. OpenTelemetry "clients". This could significantly improve debugging, especially performance.

#73 just removed the test to make CI work again

One problem I encountered making the mobile app was that the wallet module currently enables the bitcoinconsensus feature on rust-bitcoin which compiles bitcoin core at build time.

The compilation was broken in the mobile environment, and I didn't want to debug that because we only use bitcoinconsensus functionality in this unittest.

It would be nice to only compile bitcoin core in situations where we need it.

If we want to eventually build a mobile/web client for MiniMint having WASM bindings would be super useful. For that there probably needs to be a different API client that uses a WASM runtime compatible HTTP client and a different database implementation that doesn't rely on file system access but e.g. local storage.

We don't consider change right now and bleed money in fees. To fix this we need:

• deterministic change address generation (include randomness beacon in wallet consensus)
• fee-aware coin selection
• fix panic on peg-out due to the faulty assumption that everyone commits their signatures in the same (directly following) round
• recognize change after it has been confirmed/stop broadcasting tx
• CPFP on congestion

I recently watched a talk about this project on the open source stage at the miami bitcoin conference.

I got the idea that you might benefit by using fidelity bonds to choose the people who hold the keys of the server's multisig. I think it fits very well with the cypherpunk ethos of the project.

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020489.html

Let me know if you have any questions, I'm happy to elaborate.

To avoid issuing denominations with low anonymity set there should be a federation API endpoint that shows issuances per denomination. The client should use that information to choose output denominations.

Random thought: the denominations should be uniformly distributed for low amounts that everyone uses. Larger amounts would see less usage but possibly due to this fix none instead. So we should see the highest actually used denomination have the highest usage because all the theoretically higher ones would be split into this one. We could use the amplitude of that behavior to judge when to bootstrap a higher denomination.

While benchmarking I noticed that coin selection is a bottle neck since we load all coins into memory first and then run our algorithm. Coins should probably be saved by denomination instead and retrieved only on demand to increase efficiency. Should be considered in #25 too (prefix search/RocksDB families).

Strings tend to lead to confusing errors if path contains unexpected character.

Priority: low

Each module knows its assets and liabilities, the Module trait should expose these and assert that after every round assets>liabilities.

Sled seems not mature enough and development has slowed down. In the beginning I thought they might get to a stable release before MiniMint, but that's probably too optimistic.

I'd like to stick to dumb key-value stores with transactions, prefix-fetching and deterministic ordering because there seems to be less room for accidental consensus-incompatibility that way (e.g. complicated SQLite queries would be hard to analyze for being deterministic on all platforms). I'd also like the database to be embedded to reduce complexity (currently, if there's a DB error it's safe to assume something went horribly wrong, like the underlying drive unmounting, and we can shut down). That leaves a few options off the top of my head:

• LevelDB: my current favorite, used in Bitcoin Core, so already a dependency
• SQLite as a simple KV store: probably also a great choice, feels hacky though
• Berkeley DB: used in Bitcoin Core, but has caused problems historically afaik

I did not deeply evaluate any of these options and would be happy to hear opinions on these or other KV stores.

EDIT 1: I confused LMDB with LevelDB, fixed

While the prototype expects the user to pass tokens to the payee, the final version should seamlessly integrate with other mints or merchants via lightning. That means that local transfers and remote transfers ideally look the same from a user's perspective.

Sending

Every payment is uniquely identified by a LN invoice. The user wanting to pay it sends it and some tokens of higher value than the invoice to the federation. They also include some blinded tokens for change. This both avoids problems with not having coins of the right denomination and the fees for the LN payment not being known upfront (it's still good to have a field for maximum fees to pay).

Remote payment

The federation starts the LN payment (it is unclear as of yet how to best do this in a way that matches the federated trust assumptions). After it is done it reissues the amount overpaid by the user. The user fetches the reissued tokens later on. The mint-to-mint latency till payment confirmation is most likely one local consensus round + LN routing time + one remote consensus round. It depends a lot on the actual LN integration.

Local payment

If the mint detects that it would be paying itself over LN it can take the shortcut of directly reissuing tokens to the payee as well as change to the payer. The latency till payment confirmation of this mode is reduced to one local consensus round.

Receiving

This works similarily. The biggest problem is the actual LN integration it's either a federated LN node (hard to build and probably slow) or a ricardian contract between a LN node that has to post a bond with the federation and the user, which can ask the mint to enforce the contract.

Make sure certain data is written to the DB before sending transactions to the federation. Otherwise funds can be lost if a crash happens.

The MiniMint server is quite modular now, meaning that new input/output types can be added to the transaction type easily. It would be interesting to explore how to also modularize the client. It's a bit harder imo because most interesting transaction types involve different modules and thus there will always be quite a bit of cross-module code. But maybe some things can still be separated out.

Now we are using nix to install rust.

The integration test PR #117 introduced circular dependencies by adding mint-client and ln-gateway to dev-dependencies of minimint crate. As a result, "go to definition" doesn't work in VSCode w/ rust-analyzer extension (rust-lang/rust-analyzer#12407), but it always worked before #117. This seems like a failing of rust-analyzer, but given it's a very popular tool we should probably try to enable a good dev experience for those using it.

# checkout integration test pr
$ git checkout 06b157f2376c9358b13be37bb736233ef0359cca

# cyclic dependencies show up
$ rust-analyzer analysis-stats .                       
[ERROR project_model::workspace] cyclic deps: minimint(CrateId(169)) -> ln_gateway(CrateId(157)), alternative path: ln_gateway(CrateId(157)) -> minimint(CrateId(169))
[ERROR project_model::workspace] cyclic deps: mint_client(CrateId(181)) -> minimint(CrateId(169)), alternative path: minimint(CrateId(169)) -> mint_client(CrateId(181))
Database loaded:     3.42s (metadata 1.04s; build 2.09s)
...

# checkout parent commit
$ git checkout HEAD^
Previous HEAD position was 06b157f add integration tests in Rust
HEAD is now at 0bbd497 Merge pull request #94 from justinmoon/tbs-test

# no cycles
$ rust-analyzer analysis-stats .
Database loaded:     3.08s (metadata 717.77ms; build 1.53s)
...

The main transaction processing loop is currently parallelized due to some heavy computation necessary for verifying and issuing e-cash tokens. While this is a working solution it is far from elegant:

• It necessitates filtering transactions for conflicts since different scheduling could otherwise lead to different outcomes
• It prevent using database transactions effectively leading to a custom batch transaction implementation
• It doesn't achieve optimal CPU utilization since it's only transaction-granular, so one huge transaction might take longer than all other transactions combined and delay the entire consensus round

This can be fixed by not parallelizing transaction processing, but instead only the computationally intensive parts. These are:

• e-cash token validation
• e-cash token issuance

While issuance is easily offloaded to an external thread-pool once a transaction is approved, validation of e-cash tokens is more complicated.

One way to implement it would be validating all blind signatures before beginning to process the transactions, effectively building a lookup table. That could happen in parallel because signature validation is a pure function. One problem with that approach is that computation is spent on entirely invalid transactions. On the other hand transactions submitted to the consensus are normally valid since federation members check validity beforehand, so this would only be abuseable by federation members and easily detected. This DoS vector will never really be eliminated imo and even exists in the current system, probably best to cross this bridge when we come to it (secondary single-sig e-cash system?).

Improve CI runtime by caching already-built artifacts in target!

There's now a cln-maintained RPC crate, we should use it.

When running the mint_client::wallet::tests::create_output unit test with --nocapture, this is one of the tracing errors.

LN node can't open a new channel during integration tests .Link to logs

configure_me is an alternative to structopt/clap written by me with focus on these things:

• ability to combine multiple sources of configuration: files and "parts" directories, env vars, arguments
• being GNU-like (supports all combinations in arguments --foo, -f, -xyz, --foo bar, --foo=bar, -xyzf=bar, -xyzfbar
• has automated man page generation
• no stringly interfaces
• OsString/PathBuf are handled correctly
• nice error messages

It currently does not support:

• subcommands
• shell completion
• typo-detection with friendly message

As a result it may be currently more suitable for daemons than daily-used CLI commands. Especially if those daemons need to be configured with private information (which MUST NOT go to arguments).

Importance: medium

While writing the liveness proof for federated e-cash I noticed there is a situation in which f=n-t malicious members can prevent the federation from making progress on signing e-cash tokens.

Each HBBFT consensus round includes the proposals from n-fmembers, otherwise f members could block the consensus by delaying their contribution indefinitely. The other way around works too by just being early, then some other random honest member will be left out. If that's the case the dishonest member could simply never contribute signature shares and the federation would only ever get n-2f signature shares, which is less than the required t to combine to a signature.

There are two ways to fix this:

1. Requiring members to contribute signature shares the next round one is part of the consensus set and otherwise be banned (by not processing the peer's messages anymore). Unfortunately that would mean not being able to postpone signing if too little CPU resources are available (tx processing is more important, as soon as a tx is confirmed it can be accepted even without receiving e-cash tokens because that is guaranteed to happen eventually).
2. Requiring members to contribute a signature share after Δe epochs in the next round they are in the consensus set, otherwise they get excluded.

I think 1 would be more reasonable for now even though it requires some re-engineering afaik to make sure members always contribute in time. But premature optimization has bitten me way too often by now and 2 is just very comlex …

https://github.com/fedimint/minimint/blob/12bd6a5a5259b09e31ec82a77f1ed191ce173c87/modules/minimint-mint/src/lib.rs#L267

Should be greater or equal. The threshold is the minimum number of shares required to build a signature. Found it while hacking on some benchmarks: https://github.com/fedimint/minimint/blob/2022-04-benchmark-hack/benchmark/src/main.rs

The integration test should wait for the LN Gateway to actually receive the e-cash tokens for its LN payment. This probably requires creating a new (authenticated) API endpoint for LN Gateway operators to query its balance.

This was a hack when RNGs were needed during parallel processing of transactions. We can get rid of it either by having a central RNG if no parallel usage occurs anymore after #50 or replace it with thread_rng that, as I found out, also appears to be a CryptoRng.

#60 made integration tests in CI slow, maybe caching can help as it did before. Probably needs some special Nix caching.

We are testing with nix unstable in CI, is there any way to make it run on stable? Or do we just need to wait? This made debugging #77 rather annoying on my machine.

minimint$ nix-build 
error: unsupported argument 'submodules' to 'fetchGit', at /nix/store/9zfki3r0ma0vmqin3iwlbv2vmskp4jb7-naersk-src/lib.nix:117:11
(use '--show-trace' to show detailed location information)

Our JSON files currently serialize threshold_crypto::PublicKey as an array of bytes and serialize threshold_crypto::PublicKey as an array of arrays of bytes. It would be better to use hex instead of arrays of bytes to serialize these public keys.

#[serde(with = serde_binary_human_readable)] seems to correctly hex-serialize threshold_crypto::PublicKey. But I'm not sure how to serizlize threshold_crypto::PublicKeySet. It doesn't have a way to iterate over the public keys because the commit attribute is private.

We currently mostly use rand 0.6 and 0.8 for bls12-381, ideally we could switch to a common, up to date version. For this we need to:

• investigate why rust-bitcoin is still on 0.6, probably MSRV but maybe rand-core can fix some problems
• update hbbft which entails upgrading their whole crypto ecosystem

Inputs:

• last round consensus item
• current DB state

Outputs

• new consensus items
  • transient: only valid for one round
  • persistent: valid until part of consensus outcome or other invalidation criteria (invalidation interface?)
• database changes to be applied atomically before next consensus round (ideally accumulated using mut ref to avoid many small allocations)

LND is too popular to be ignored at this point. For a while I was thinking about creating a crate with set of types and traits common to all LN implementations, which could abstract over them. I'm interested in collaborating here.

I made tonic_lnd crate but it's unfortunately for tokio only and this crate uses async_std. Maybe finally a motivation to create a unified trait for them. :D

Priority: medium

The MiniMint server is quite modular now, meaning that new input/output types can be added to the transaction type easily. It would be interesting to explore how to also modularize the client. It's a bit harder imo because most interesting transaction types involve different modules and thus there will always be quite a bit of cross-module code. But maybe some things can still be separated out.

The docs for running the federation for testing are outdated.

How tolerant does the wallet module need to be relative to input errors and RPC errors of all kinds? There are many expect calls throughout the code. Do we prefer crashes or are these temporary measures to move development forward?

Some of the changes MiniMint needs have landed upstream by now, check if we can get rid of some of our forks of:

• rust-secp256k1[-zkp]
• bitcoin_hashes

for example in this run

in unit test: just compiles the workspace crates and all the deps are used from cache
in integration test: every crate is being compiled

Add a test that pegs in, reissues and pegs out using regtest bitcoind.

In #71 (and also during some hacking with @justinmoon) I rediscovered the hard way that I didn't think about parallelism when building the client since it was only ever used in CI tests. This needs to be addressed to build robust clients. One way to do so would be using transactions that will be introduced in a fix to #25. A short term fix are Mutexes as discussed in #71 (comment).

Use ElementsProject/lightning#5012 to make out LN gateway run as a plugin. This will remove the cln rpc socket path from the config and will enable intercepting HTLCs for incoming payments.

The current network stack is fragile. If a connection dies the node goes down, killing all other node. This is obviously not viable for production. Instead a network stack that

• attempts to reconnect,
• caches messages that were not ACKed by the other side and
• uses Tor

would be nice.

After the transaction refactor there is now the possibility to have change outputs that issue mint tokens back to the sender. This means that the client does not need to fail if an amount can not be represented with tokens in possession. Instead larger ones can be used and change returned to self.

We should expect peers will temporarily disconnect (e.g. for upgrades or network outages) from consensus. In order to support rejoining, peers will need to have an API for downloading the consensus history. Additionally this could serve as a mechanism for users to privately download their coins if the nonces and blinding keys were lost (and generated deterministically from a known secret key).

Some of the features of consensus download should be:

• Epochs should be signed so one can verify that a quorum of fed members agree on the history
• Epochs should include a last_non_empty_epoch or something similar so that empty epochs do not need to be downloaded
• Rejoining nodes can specify the epoch from which they need to begin a download

We might also want to put epochs into a merkle tree so if ever consensus history was rewritten we could more easily verify that an epoch is in the current consensus, although I'm not sure if that scenario would ever occur.

When I tested spend + reissue I encountered "FinalizationError(UnknownIssuance)", and then it worked again without restarting (federations, clientd) or changing anything. The reissue status was Accepted though :

It should be possible for users to easily choose which modules they want to use at compile time. Ideally users should be able to add modules from foreign crates.

I see 4 solutions of which 3 also fulfill the latter requirement::

1. Feature Flags: All modules have to live in this repository but can be activated via feature flags
2. Macros: Expose macros that take modules as input and build all the enums (transactions, inputs, outputs etc.) with one variant per module.
3. Generics Hackery: We could emulate variadic generics to a degree by having MinimintConsensus variants with 2 to e.g. 6 type parameters, each being a module (impl via macro). Alternatively (and even cooler) we could use advanced generics like heterogenous lists.
4. Dynamic Modules: Make the Module trait object safe and work with Box<dyn Module> objects. I like this idea the least since it would remove a lot of the guarantees Rust's type system gives us.