bitcoindevkit / bdk Goto Github PK

I suggest adding this to the roadmap on the website.

Inspired by this tweet https://twitter.com/matt_odell/status/1286659125322162176?s=19

"All wallets should have the option to use your own node easily by simply scanning a QR code or pasting a connection code."

The BDK can build this and make it easy-to-use / default for any wallet based on BDK.

Create example code and blog post for how to use integrated rust tor client to sync bdk wallet with tor enabled blockchain service such as blockstream.info (http://explorerzydxu5ecjrkwceayqybizmpjjznk5izmitf2modhcusuqlid.onion/).

see:

https://github.com/MagicalBitcoin/libtor
https://gitlab.torproject.org/tpo/core/arti

This is an overview issue (based on same issue for murmel) for discussion and tracking the addition of a fully functional compact filters Blockchain backend to bdk. Ideally we'll have additional issues and/or PRs for each missing feature to link to this issue and we check off each feature as it's completed.

The below BIP-157/158 protocol features are based on the BIP-157 Client Operation section. Please suggest additions and corrections in the comments.

Missing protocol features for bdk are indicated below with an unchecked box [ ], completed features with a checked box [x].

Terminology

• peer: bitcoin full node peer
• block header: bitcoin standard block header
• filter header: BIP-157 compact block filter header
• block filter: BIP-157/158 compact block filter

Peer Discovery

1. • Allow user to specify peer addresses
2. • Receive peer addresses via [addr] messages from connected peers
3. • Store new peer addresses
4. • If no provided or previously stored peers, find seed peer addresses via DNS

Connect to Peers

1. • Allow user to specify number of peers to connect to
2. • Allow user to specify minimum number of BIP-157 peers to connect to
3. • Connect to peers and request signalling bits via [version] message
4. • Keep track of peers signalling service bit NODE_COMPACT_FILTERS
5. • Connect to specified number of peers
6. • Connect to minimum number of BIP-157 peers

Request Block Headers

1. • Request block headers via [getheaders] message to connected peers
2. • Request block headers from genesis to most work block, 1 to 2000 blocks per request
3. • Optional: Request block headers from stored checkpoint block instead of genesis block
4. • Optional: Request block headers from DNS (see https://bitcoinheaders.net/)

Receive Block Headers

1. • Receive block headers [headers] from peers
2. • Build chain of block headers from stored and received new block headers
3. • Store new block headers and new block header tip
4. • Disconnect peers whose longest chain has significantly less work than known tip
5. • Request block filter headers from all peers for any new block header received

Request Filter Headers

1. • Allow user to specify start block height for block filter sync
2. • Request filter headers via [getcfheaders] message to peers
3. • Request from start block height to end block hash (1-1000 per request)
4. • Request filter headers for the same block from more than one peer
5. • Optional: First request filter headers at intervals of 1000 via [getcfcheckpt] message
6. • Optional: Request filter headers from DNS (see https://bitcoinheaders.net/)

Receive Filter Headers

1. • Receive filter headers [cfheaders] from peers
2. • Verify filter headers from multiple peers are the same
3. • If any received filter headers for a block are different,
   1. • request full block and compute correct block filter and filter header
   2. • ban peers that provided incorrect filter header
4. • Build chain of filter headers from stored and received new filter headers
5. • Store verified filter headers, at least to depth of 100
6. • Request block filter for any new verified filter header
7. • Optional: store more than 100 verified filter headers

Request Block Filters

1. • Request block filters after all filter headers are received and verified
2. • Request block filters from start block via [getcfilters] message to peers
3. • Request from start block height to end block hash (1-1000 per request)
4. • Optional: Check if filter header is for empty block filter if so skip request
5. • Optional: Request different block filters in parallel from multiple peers

Receive Block Filters

1. • Receive block filter [cfilter] from peers
2. • Verify block filters sequentially, must link to corresponding filter header
3. • Ban any peer that sends an invalid block filter
4. • Check block filters for relevance
   1. • Previous output scripts for transaction inputs
   2. • ScriptPubKey for transaction outputs

Fetch Full Blocks

1. • If fetching full blocks, download from random peers
2. • Fetch full blocks from peers even if they don’t support compact block filters
3. • Fetch full blocks that correspond to relevant block filters

Add a DNS based block data source to fetch block headers and block filters as described here: https://bitcoinheaders.net/

This would provide an alternative source of block data and way for a wallet to detect network censorship.

• Secure enclave key storage and signing
• Hardware wallet / HWI
• HSM runtime support like the nShield

This may need to be moved or companion issues created in bdk-jni (and future bdk-ios) repos

Example:

use magical_bitcoin_wallet::descriptor::ExtendedDescriptor;
use std::str::FromStr;

fn main() {
    let desc_str = "wsh(multi(1,[9d033c9d/48'/1'/0'/2']tpubDEHKJA6XKc6QaRvc5gunJY9T9cst9m1y8o2owqrvUmXK69ZjWySB7hgB6HvFdTp5MamaKYumSGxL49VUiEjWrhDE8N8BympFWDknLBEPG38/0/*,[6bb3d403/48'/1'/0'/2']tpubDDw2d4rKFtn5ECCq4KQ6GTi73YUQt3Wz2gqyv19kLAPJQKKKzNvEESi9v44sG4Fc6DsBuDgo5FQ9ZxbzQfV2X7jRvsBcT7N5s6g57xS4aUM/0/*))";
    ExtendedDescriptor::from_str(&desc_str).unwrap();
}

Output:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Miniscript(Unexpected("multi(3 args) while parsing Miniscript"))', examples/magical.rs:6:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Here is where the error is being created in rust-miniscript. You can see a couple lines above that is where multi should be handled, but it seems there hasn't been a rust-miniscript release since it was added.

See also: https://github.com/MagicalBitcoin/magical-bitcoin-wallet

https://github.com/rust-bitcoin/rust-bitcoincore-rpc

Two related privacy features I'd like to see implemented in the future:

1. per-UTXO coin control during transaction construction
2. blacklisting of individual UTXO(s) to combat dusting attacks

BDK should be able to provide blockchain and key services for the Lightning Development Kit (LDK). The LDK traits that need to be implemented by BDK are:

• lightning::chain::keysinterface::KeysInterface
• lightning::chain::chaininterface::BroadcasterInterface
• lightning::chain::chaininterface::FeeEstimator
• lightning::chain::Filter
• lightning::chain::Confirm
• lightning::chain::Access
• lightning::chain::Watch

Some notes on providing support for lightning via EPS, but also relevant for making sure BDK can support Lightning clients.

chris-belcher/electrum-personal-server#174 (comment)

Probably something like a "AnyBlockchain" type defined as an enum that internally matches the blockchain variant.

Ideally it should be built using Box<dyn Blockchain> but I'm not sure if those traits are object safe

bwt is a lightweight wallet xpub tracker and query engine for Bitcoin, implemented in Rust.

Implement block data source trait for bwt server REST API.

depends on #84

We are currently discussing in rust-bitcoin how to model the PSBT workflow in rust. Bdk seems to be the most advanced rust wallet implementation so far. Do you have any opinion on rust-bitcoin/rust-bitcoin#457 for example? From what I read in the bdk docs I figure that my wallet trait might be too simple and not capturing UTXO selection parameters very well (something I thought to be intrinsic to a certain wallet impl). Your Signer trait also seems more adapted to the real world as it differentiates between signing single inputs and whole transactions at once. Given your experience, do you think unifying the interface for PSBT handling is feasible (so that e.g. new HW wallet crates can be compatible with all PSBT aware rust software)?

Suggestion from @moneyball. Create a library that allows a light client to use the p2p network to monitor the mempool. today, the only way you can do this is with Core, which implies running a fully validating node. however, there is no reason you need to do IBD or validation in order to just monitor the mempool. why would a light client want to monitor the mempool?

1. self sovereign improved fee estimation
2. self sovereign watchtower (for LN or various other L2/custody protocols that have been proposed which require monitoring for certain txn and responding)
3. perhaps certain analytical tools (would be awesome if devs could focus on the analytical tool and utilize BDK to deal with all the details vs. write from scratch)

https://github.com/bitcoin/bips/blob/c1459ff17aa68a7a6e724d6b3b08b6878266e538/bip-0125.mediawiki

See also
rust-bitcoin/murmel#37

• Test the ToWalletDescriptor trait
  • From &str with and without checksum appended
  • From &str with keys from the right/wrong network
  • From the output of the descriptor!() macro

Moved to #193
- [ ] Test the ExtractPolicy trait
- [ ] Simple descriptors like wpkh(), sh(multi())
- [ ] Both with extended and single keys
- [ ] More complicated descriptor that contains timelocks too in a thresh()
- [ ] Mixed timelocks should fail
- [ ] Multiple timelocks of the same type should be correctly merged together

• Test the get_checksum() function. It should return the same value as Bitcoin Core's getdescriptorinfo

• Test the descriptor!() macro

  • Ideally at least one of each "type" of operator should be tested (i.e. one modifier, one "leaf_opcode", one "leaf_opcode_value", etc)
  • Test mixing up key types that implement ToDescriptorKey in multi() or thresh()
  • Test that the valid_networks returned is correctly computed based on the keys present in the descriptor
  • Test that the key_maps are correctly merged together
  • Test that the ScriptContext is correctly validated (i.e. passing a type that only impl ToDescriptorKey<Segwitv0> to a pkh() descriptor should throw a compilation error

• Test the existing descriptor templates, make sure they are expanded to the right descriptors

something like

pub trait ConfigurableBlockchain: Blockchain {
    type Config: Debug + Serialize + Deserialize;

    fn from_config(source: Self::Config) -> Result<Self, Error>;
}

... and the same for the database

depends on bitcoindevkit/bdk-old#18
will support bdk-android project

https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki#Introduction

https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_5):_Initial_Block_Download#Checkpoints

Generally need to think about how to prevent eclipse attacks for light clients that sync block headers from peers. Some ideas of strategies that can be borrowed from Bitcoin Core:

• general core heuristics: feeler connections, new/tried nodes tables, test-before-evict, etc.
• using ASMAP table information (https://bitcoincore.reviews/16702.html)
• reverse header lookup strategies and banning peers that give bad headers

depends on bitcoindevkit/bdk-old#18
will support bdk-ios project

For backing up wallet and seed data
See also https://github.com/photon-sdk

Seeing as you guys are a descriptor based wallet it would be really cool if you could support this wallet export/import format: https://github.com/Fonta1n3/FullyNoded/blob/master/Docs/Wallets.md

It is quite simple and is made possible by directly relying on Bitcoin Core descriptors. The functionality is live on Fully Noded and Specter-Desktop, hopefully more wallets will start to use this format as it offers an excellent way for users to backup, export and import wallets.

rust-bitcoin/rust-wallet#17

• manual
• branch and bound

electrs is a bitcoin block chain index engine and HTTP API, implemented in Rust.

Implement block chain data source trait for electrs server REST API.

depends on bitcoindevkit/bdk-old#19

• Local SQLite
• Local Flat file
• Cloud Encrypted flat file (Backup)

Generate new random BIP-39 mnemonic codes. Allow 12, 18, or 24 word codes.

Create BIP-32 extended private keys (xprv/tprv) from BIP-39 mnemonic codes OR from a randomly generated seed.

Create extended public keys (xpub/tpub) from corresponding extended private keys.

https://github.com/MagicalBitcoin/rust-electrum-client

this is a place holder for alternative transports like:

• low bandwidth
• headers over dns
• etc.

Also make sure that they are run with https://doc.rust-lang.org/rustdoc/documentation-tests.html#include-items-only-when-collecting-doctests

Test all the backends with the #[bdk_blockchain_tests] proc macro like we do with electrum in the CI. This could be done while transitioning to GitHub Actions (see #59).

I would use nigiri to have an electrs/esplora local instance, then a btcd node for compact filters.

Currently the CI downloads and runs the full node but that's pretty slow, we could make docker containers with everything preinstalled for it.

A docker image for electrs was added here: 98803b2

Seems to be much better integrated than travis

I suggest adding this to the roadmap on the website.

Inspired by this tweet https://twitter.com/matt_odell/status/1286659125322162176?s=19

"All wallets should have the option to use advanced coin control with labeling."

The BDK can build this and make it easy-to-use / default for any wallet based on BDK.

Another contributor suggested this as inspiration too: https://twitter.com/SifirApps/status/1269970305977585665

In several of the more cutting edge Bitcoin protocols you end up in a situation where there is a 2-of-2(A,B) where Alice and Bob own the two keys respectively. Later on, one of the parties (let's say Alice) learns the secret key for B through the execution of the protocol. Now Alice owns the output but needs to store and recall the secret key for B and the details of the output.

I was thinking it would be nice for magical to manage this for me. What I think I want is to be able to add some kind of auxiliary descriptors to a wallet. The wallet could store it in its database and treat it like any of its other unspent outputs. Here are some considerations that come to mind:

1. You would probably want a coin selection policy that spends these higher priority than outputs derived from your main descriptor (because they are lost if you lose database backup).
2. When doing things like get_balance you're probably going to want to see how much of your balance is derived from the main descriptor and how much is from these transient descriptors. Likely you will want to "sweep" all of these in one go back to the main wallet descriptor.
3. Sometimes these outputs have a pending tx that may spend from it in the future (e.g. revoked lightning commitment transactions) so the user needs to spend from it right away. Although it should be the application's job to make sure this happens, it would be cool if you could mark the auxiliary descriptor with a block height or time for which it must be spent before.

I was thinking about trying to implement something like this myself but wanted to check if you had an existing vision for this.

This is done manually now to have the ability to return "offline" instances for them (Blockchain::offline), but it's complex and repetitive.

Starting with an empty ~/.magical-bitcoin, it crashes if we provide a descriptor containing a checksum:

$ magic --descriptor "pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)#gn28ywm7" policies
...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ChecksumMismatch', examples/repl.rs:71:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Furthermore, a checksum for the entire checksummed descriptor is stored in sled, which causes a conflict when we try to provide the same valid descriptor without a checksum.

$ magic --descriptor "pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)" policies
...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ChecksumMismatch', examples/repl.rs:71:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If we delete ~/.magical-bitcoin and re-run the last command it will work.

I believe this line should check to see if the descriptor already contains a valid checksum.

This is a tracking issue for other issues and PRs related to supporting Taproot in BDK.

Dependencies

Overall rust-bitcoin org tracking project

rust-bitcoin/rust-bitcoin

• rust-bitcoin/rust-bitcoin#503

rust-bitcoin/rust-miniscript

• rust-bitcoin/rust-miniscript#267
• rust-bitcoin/rust-miniscript#275
• rust-bitcoin/rust-miniscript#278
• rust-bitcoin/rust-miniscript#182

Send to Single Signature TR Address

• #402
• #394

Create Single Signature TR Descriptor Wallet

• #463

Create Multi Signature TR Descriptor Wallet

Syncing again seems to fix it