Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

In order to reproduce the issue, follow these steps:

1. I stress test 2pc architectrure on 8 real machine for 8 sharding,each machine runing one sentinel ,coordinator,locking_shardd
2. runing ./tools/bench/twophase-gen and Statistical throughput count from tx_samples_x.txt,and find the throughput not stable ,as show below ,the first cloumn is count ,the second cloumn is timestamp,why?
   42 1644890386
   43724 1644890393
   119916 1644890394
   30255 1644890395
   13765 1644890396
   2840 1644890397
   2 1644891076
   3 1644891077
   1 1644891101
   8 1644891107
   15 1644891170
   13 1644891172
   12 1644891173
   1 1644891459
   2 1644891473
   1 1644891474
   13 1644891529
   18 1644891540
   3 1644891541
   1 1644891637
   7 1644891638
   4 1644891661
   42 1644891684
   2 1644891826
   3 1644891828
   51904 1644892939
   39527 1644892940
   13126 1644893569
   23061 1644893570
   3549 1644893571
3. The CPU usage of the core where a process coordinator resides and the core where process locking_shardd resides reaches 100%, but other cores are idle. whether CPU binding is used?

Code of Conduct

• I agree to follow this project's Code of Conduct

Abstract

I had some doubts when I read the paper recently. But there is no contact information in the paper, so ask a question here, hope to get the answer, thank you!

1. efficacy of shard
2. doubt of preventing double spends

Description

1. When all input UHS are in one shard, will this transaction send to atomizer? If so, how can user get result form watchtower; if not, atomizer should handler all transactions, how shard improve performance.

2. Only after shard get result of one transaction form atomizer, it can process another transaction? If so, it will be so inefficient. If not, send transaction with one UHS two consecutive times. When shard get these two, inputs are validate both.

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

I'm trying to build via Docker on Mac Pro (Apple M1 pro). The docker build continues to fail at the point of cloning the asio repo:

#8 2.959 + cd /NuRaft-1.3.0
#8 2.959 + '[' '!' -d asio ']'
#8 2.959 + git clone https://github.com/chriskohlhoff/asio.git ./asio
#8 2.961 Cloning into './asio'...
#8 17.22 error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.

Stack overflow shows this as an error specific to the version of git using the gnuTLS. Recommends rebuilding git with openssl. Has anyone else ran into this problem?

In order to reproduce the issue, follow these steps:

1. Follow the docker build instructions in the README

Code of Conduct

• I agree to follow this project's Code of Conduct

To further ensure that a malicious actor sending specially-crafted transaction messages cannot easily produce hashmap hash collisions (leading to performance degradation and potential DoS), we should initialize const_sip_hash with random parameters (the set of possible collisions is then randomized on each system run).

(Note that SipHash is not used for cryptographic purposes, e.g., when calculating UHS identifiers; we use SHA256 there.)

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 1 1
[2022-02-13 03:59:18.147] [WARN ] Existing wallet file not found
[2022-02-13 03:59:18.147] [WARN ] Existing client file not found
fdcc8e5559af55040a4f0f12db2b4de5f614ffaf08d359d727b3b07c12a6c64f
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool1.dat wallet1.dat newaddress
[2022-02-13 03:59:54.151] [WARN ] Existing wallet file not found
[2022-02-13 03:59:54.152] [WARN ] Existing client file not found
usd1qpsctc3vzrwa2e2ru0lehgjh8p5l8y2cklsuh6sd6grr3n6eqayn7r9zgq8
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat send 1 usd1qpsctc3vzrwa2e2ru0lehgjh8p5l8y2cklsuh6sd6grr3n6eqayn7r9zgq8
tx_id:
f75737d445cdf02ba8ca7cf6372930cc974c7e9cbc2ef3fc814b379772cdecf5
Data for recipient importinput:
f75737d445cdf02ba8ca7cf6372930cc974c7e9cbc2ef3fc814b379772cdecf50000000000000000e5ad0c7243ec63ff655e959f7ed68be9eebec3c1b38e1c11956a37207956f5f00100000000000000
Sentinel responded: Confirmed
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool1.dat wallet1.dat importinput f75737d445cdf02ba8ca7cf6372930cc974c7e9cbc2ef3fc814b379772cdecf50000000000000000e5ad0c7243ec63ff655e959f7ed68be9eebec3c1b38e1c11956a37207956f5f00100000000000000
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat importinput f75737d445cdf02ba8ca7cf6372930cc974c7e9cbc2ef3fc814b379772cdecf50000000000000000e5ad0c7243ec63ff655e959f7ed68be9eebec3c1b38e1c11956a37207956f5f00100000000000000
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool1.dat wallet1.dat sync
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat sync
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool1.dat wallet1.dat info
Balance: $0.01, UTXOs: 1, pending TXs: 0
root@2af70284b1b5:/opt/tx-processor# ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat info
Balance: $0.01, UTXOs: 1, pending TXs: 0

Code of Conduct

• I agree to follow this project's Code of Conduct

We've had several previous conversations exploring some performance improvements to the system's network communications.

One avenue we landed on as a possibility was using UDP multicast to broadcast blocks from atomizers to their shards. It seems as though there may be better options for improving performance in the near-term, but this is still an option worth potentially exploring in greater detail.

How does a user learn that they have received a payment?

This appears to be a hard problem when users can't self-validate the entire state of the system (e.g. are light clients or, in our case, CBDC users who simply don't get access to all transactions) and system provides strong privacy.

In particular, in Zerocash/Zcash-style systems a user is tasked with decrypting every payment and see if the decryption succeeds. This is also true for decoy-based privacy like CryptoNote/Monero. The issue appears inherent -- strong privacy mandates that no third party can detect when a payment is sent to you.

Some potential solutions include:

• Letting participants observe the entire traffic and do payment decryption themselves. This is unworkable for our throughput and usability requirements.
• Having payment detection nodes that learn this information on user's behalf. This is, e.g., how some Zcash/Monero light clients work.
• Using fancy cryptography (e.g., FHE) to have the server compute an encrypted version of which payments are for which user, that only the particular user can later decrypt.
• Using something like Vuvuzela's dead drops to which sender will address a message.
• Simply making recipient's address available to the CBDC servers. I.e. everyone learns that a payment was made to Bob, but not that it came from Alice or how much she sent.
• Changing the transaction model: mandating that a recipient has to counter-sign to receive transaction (so by definition they know they have received money). This implies interaction but might be useful for achieving certain policy goals (e.g. in counter-signing model I can't be implicated for receiving a payment I did not consent to receive, whereas in traditional system everyone (including people I don't want to!) can send me a payment, provided they know my address)

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

A few header files have extraneous /return commands causing Doxygen to fail. These appear to be innocent typos due to copy/paste.

In order to reproduce the issue, follow these steps:

1. Clone repo

git clone --recurse-submodules https://github.com/mit-dci/opencbdc-tx

2. Run Doxygen

doxygen Doxyfile

The errors you'll eventually see are:

...
Generating call graph for function cbdc::transaction::wallet::seed
/Users/duc/Dev2/opencbdc-tx_dmt/src/uhs/transaction/wallet.hpp:142: error: documented empty return type of cbdc::transaction::wallet::seed_readonly (warning treated as error, aborting now)
...
Generating call graph for function cbdc::watchtower::error_cache::check_tx_id
/Users/duc/Dev2/opencbdc-tx_dmt/src/uhs/atomizer/watchtower/error_cache.hpp:36: error: documented empty return type of cbdc::watchtower::error_cache::push_errors (warning treated as error, aborting now)
Exiting...


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

Testing outside of Docker may fail if a port number is needed for the test and the port is in use. In some cases, this may result in a seg fault which can be misleading. For example, the unit tests in raft_test.app create 3 nodes using the port range 5000-5002. The port range has caused seg faults on my machine as port 5000 was in use. Changing the ports resolves the issue.

I'm not sure what the best fix for this is. Finding available ports (in code) can be a little tricky and unreliable. The other alternative is to just test in Docker and specify that testing outside of Docker is not recommended....

Code of Conduct

• I agree to follow this project's Code of Conduct

It turns out that we can't do much inside a signal handler. Some of our handlers lock mutexes and notify condition variables, which is not allowed by the C++ spec. cppreference includes a good explanation of the limitations on signal handlers.

SHA256, currently used, has a number of points one could improve upon:

• it does not protect against length extension attacks (not currently relevant for our work). This is cited as a reason why Bitcoin uses double-SHA.
• it does not support explicit personalization fields for domain separation (so a solution for #5 is to use H(H(tag) || H(tag) || data) for SHA256).

While SHA256 design has been around for a long while, similar designs SHA0 and SHA1 are broken, and cryptographic community's hash function design techniques have improved in the 20 year period between designs of SHA256 and SHA3.

In this proposal, we would still keep SHA256 for Schnorr signatures as libsecp is a highly complex codebase we don't need to edit but could use SHA3 everywhere else. Most NIST PQC signature candidates also rely on SHA3 so it is likely we would bring it at some point in the future.

Refactor atomizer::atomizer_raft and atomizer::state_machine to use raft::rpc::server and rpc::client.

In the current UHS-based data model, sentinels validate the transaction-local invariants of a full transaction, before converting the transaction to a compact representation which is the data structure processed by the backend (2PC or Atomizer). Since the data required to validate the full transaction is lost after being converted to a compact transaction, it would be trivial for a compromised sentinel to submit an invalid transaction for processing. This would allow the adversary to spend any UHS ID, and mint arbitrary UHS IDs representing any value (that an honest sentinel would later accept as a valid output). This is possible because the upstream component from the sentinels (shards in the atomizer architecture, coordinators in 2PC) blindly accepts compact transactions (deleting input UHS IDs, and creating output UHD IDs) and has no knowledge of the pre-image data. The current implementation does not restrict the origin of compact transactions either, so any entity with internal network access can send compact transactions directly to backend components, bypassing the sentinels.

It would be desirable to protect against up to n sentinels being compromised. n should be configurable based on the threat model and number of nodes in the system. As long as <=n sentinels are malicious, it should be impossible to settle an invalid transaction. Furthermore, upstream components should only accept compact transactions for processing from known sentinels.

Question

What are the methods and trade-offs of improving the system's ability to detect an attacker inflating (or deflating) the monetary supply?

Benefit

Tamper-detection is a stepping stone toward tamper-prevention (which is an important area of CBDC research). The system as-implemented now has no method of detecting if any machine has been compromised or if a bug has caused incorrect data to be written in the UHS (specifically, an attacker compromising a single machine is able to undetectably alter the monetary supply). For example, a shard can insert a malicious UTXO in its state and continue system operations as normal. This inflates supply, yet we don’t have any detection or prevention mechanisms against this. There are several ways to potentially improve the system's ability to detect such an attack, but the trade-offs are unclear (performance degradation seems possible, at least). Knowing the trade-offs, inherent or not, in various methods of tamper-detection may inform practical endeavors and potentially reveal other avenues to explore.

Proposed Solution

As the system currently offers no detection ability, the first identified threat model to address is to allow the system to detect tampering (even if after-the-fact) in the face of a strong attacker achieving total compromise of any single machine. (In this proposal, we don't consider weaker attack models where the attacker only has access to a single node’s storage layer or cryptographic keys, etc., nor do we consider the case where a problem in a component would cause multiple nodes to be compromised, like a bug in LevelDB.) This threat model only focuses on detecting the alteration of the monetary supply after such a compromise occurs. This is weaker than protecting against multi-machine compromise, but it is a first step towards a more hardened system. Other threat models and solutions should be explored further in other proposals.

There are a variety of methods to explore which would enable solving for the above threat model. Of the ideas generated, the options below stand out as avenues to explore first (though they are not necessarily the best options and need to be evaluated against the multiple goals our system has).

Guarding against a sentinel compromise:

• #84

Guarding against shard/atomizer/coordinator compromise:

• #100
• #101
• Add an automated tool to run a supply audit

Possible Difficulties

There are already trade-offs between storing values in the UHS and leveraging more cryptographic techniques: in particular, storing values in the UHS, though perhaps the simplest solution, goes against one of the purposes of using the UHS in the first place (blinding values from the transaction processor), and cannot meet the threat model if the compromise takes place during an audit; however, leveraging cryptographic techniques (e.g., Pedersen commitments, range proofs, etc.) may come with significant performance decreases (though it preserves the spirit of privacy that the system currently possesses).

Prior Work

• Pedersen, Torben P.. “Non-Interactive and Information-Theoretic Secure Verifiable Secret Sharing.” CRYPTO (1991).
• Bünz, Benedikt et al. “Bulletproofs: Short Proofs for Confidential Transactions and More.” 2018 IEEE Symposium on Security and Privacy (SP) (2018): 315-334.
• Narula, Neha et al. “zkLedger: Privacy-Preserving Auditing for Distributed Ledgers.” IACR Cryptol. ePrint Arch. (2018).
• Tamassia, Roberto. “Authenticated Data Structures.” ESA (2003).

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

mit-dci/opencbdc-tx

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

In order to reproduce the issue, follow these steps:

1. Modify a source file : /opencbdc-tx/src/uhs/client/client-cli.cpp

2. Issue command : docker build . -t opencbdc-tx (make sure that it compiles fine)

3. Issue command: docker-compose --file docker-compose-atomizer.yml up
   

4. Issued command : docker run --network atomizer-network -ti opencdbc-tx /bin/bash
   This will switch dir to: /opt/tx-processor

5. Check the source file located in this dir: /opt/tx-processor/src/uhs/client/client-cli.cpp
   This source file is still the original unmodified file, and does not contain the changes in step (1). This may be the reason the client-cli binary is unaffected by the changes made in (1).

Probably a docker syntax I may be getting wrong. But wanted to check it out here..

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

After launching the system, run the following:
./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 1000 2000

The above will try to mint one thousand (1000) $20 dollar bills. After submission, this transaction will be stuck in pending state and never complete. This happens when minting to a new wallet or an existing wallet.

Code of Conduct

• I agree to follow this project's Code of Conduct

This came up in a discussion with @madars yesterday.

Using BLS signatures rather than Schnorr would allow for non-interactive signature aggregation. The benefit (which may not be worth the effort, depending on the make-up of real-world transactions) would be that a single witness per-transaction would suffice, rather than per-input as-is. Additionally, verification would be one-per-tx rather than per-input.

Obvious downsides are additional complexity and reliance on less-studied crypto primitives and assumptions.

From https://crypto.stanford.edu/~dabo/pubs/papers/aggreg.pdf:

"Consider a set U of users. Each user u ∈ U has a signing keypair (PKu,SKu). We wish to aggregate the signatures of some subset U ⊆ U. Each user u ∈ U produces a signature σu on a message Mu of her choice. These signatures are then combined into a single aggregate σ by an aggregating party. The aggregating party, who can be different from and untrusted by the users in U, has access to the users’ public keys, to the messages, and to the signatures on them, but not to any private keys. The result of this aggregation is an aggregate signature σ whose length is the same as that of any of the individual signatures. This aggregate has the property that a verifier given σ along with the identities of the parties involved and their respective messages is convinced that each user signed her respective message."

Sharing a couple ideas I had on changing the architecture to simplify transaction protocols for users

1) Simplify confirmation process for wallets by shifting responsibility to the sentinels: In both architectures, any querying of the watchtower could be shifted to the sentinel. Sentinels would be a one-stop stop for users submitting or wanting to know the status of a transaction. For the submitter, the sentinel would keep the client connection open (same as 2PC currently) and would query the watchtower within a time-out window to check its status. It could similarly query for or track any error messages if the transaction fails. For the receiver, the sentinel would perform the same function by forwarding their query to the watchtower. Sentinels could scale in parallel to accommodate this functionality without any major impact on system performance

2) Solve the output sharing problem while maintaining privacy by having temporary (<5 minute) transaction storage at the sentinel layer. Our system requires out-of-band communication of outputs despite this data being seen by the sentinel. Recipients (or whoever doesn't submit the transaction) could sign a transaction status query to the sentinel containing their public key and a signature proving their identity. The sentinel layer would either direct the query to the correct sentinel or some temporary transaction data storage service, which would match the query's public key with recent transactions, including the outputs, transaction ID, and status information. The sentinel layer would delete transaction data either after one or both public keys have queried it OR at an upper bound of X minutes. This prevents long term data storage that would have privacy implications, isolates the performance impact, and simplifies the transaction protocol for users (likely improving usability)

Question

In the Phase 1 paper of Project Hamilton we concluded that "The atomizer itself is the limiting factor for the overall system as all transaction notifications have to be routed via the atomizer Raft cluster.". Where lies the exact bottleneck of the Atomizer, and can it be alleviated?

Benefit

By knowing where the exact bottleneck of the Atomizer lies, we can look at potential solutions to make the Atomizer perform better, or make informed recommendations about the infrastructure on which the Atomizer can perform in the most optimal way. For instance: if the problem lies in bandwidth between Atomizer RAFT cluster participants, we can recommend having dedicated, high bandwidth low latency network connections between them. If the problem lies in the RAFT log storage, we can recommend using solid state storage. And so on.

Proposed Solution

We made a local benchmarking setup that's isolated in a separate branch. The documentation in that branch includes how to run the benchmark here.

Our initial approach was to start with just logging the event of receiving the transaction, and not further process it. This lead to a peak of about 350k TX/s. We then began moving this point of discarding the transaction further in the atomizer's process. The transaction processing follows the following code paths:

1. Transaction is received by cbdc::atomizer::controller::server_handler
2. Transaction is received by cbdc::atomizer::atomizer_raft::tx_notify

• This method is called by all individual shards, with attestations for their respective input
• This method will place the transaction in a mutex guarded vector of complete transactions m_complete_txs here

3. Transactions are compiled into batches by cbdc::atomizer::atomizer_raft::send_complete_txs

• This method will make an aggregate_tx_notify_request containing multiple tx_notify structs and send them to the raft state machine

4. Batches of tx_notify are then received and processed by cbdc::atomizer::state_machine::commit
5. For each transaction, cbdc::atomizer::atomizer::insert_complete is called to verify the attestation being within the STXO Cache Depth (thus not expired), and none of the inputs are in the STXO cache (spent)

• If all these checks succeed, the transaction is added to the STXO cache, and to a vector of transactions that should be included in the next block, m_complete_txs, here.

6. The transaction is then included in the block by the swapping of the m_complete_txs vector that happens in cbdc::atomizer::atomizer::make_block here
7. The block is then returned to the cbdc::atomizer::state_machine::commit function here
8. The block is then returned to the raft callback in cbdc::atomizer::controller::raft_result_handler here where it's distributed to the watchtowers and shards

We have been moving the point where the transaction gets discarded further down this code path. The point where we are currently, is where the block and error data are returned from the RAFT state machine (which is after point 7. What we return from cbdc::atomizer::state_machine::commit is passed to the callback function of bullet point 8).

From the baseline test, commit 9fa80fb, we disabled block transactions and errors being returned from the raft state machine commit 01e578c.

This elevates the peak throughput of the Atomizer from 170k TX/s to 250k TX/s. But errors are no longer reported to the watchtowers and blocks only contain the height.

The assumption after this analysis is that there is a bottleneck in either the RAFT serialization (here) / storage (here) or in the callback functions processing the return value (here) and broadcasting them to the shards and watchtowers.

Further analysis is needed to pinpoint the exact problem and then we can work on a solution that resolves this.

Possible next steps

Things that can be tried to further pinpoint the issue:

• Either of these two paths can be investigated:
  1. Look at the efficiency of RAFT serialization and storage, replace with an optimal implementation:
  • Replace the RAFT Log store with an in-memory version to rule out the RAFT storage from being the bottleneck;
  • Look at the efficiency of the RAFT serialization (specifically the use of a temporary local buffer)
  2. Side-step the impact of RAFT serialization for return values by minimizing the return value size (for instance: store blocks in a member variable of the atomizer class and fetch them from the controller, and just return the height in stead of returning the whole block from the state machine)
• Look at how the network stack impacts performance, specifically when sending and receiving lots of data at once. The experiment in commit 01e578c shows the difference between full blocks and empty blocks being sent over the network - so still the same amount of block objects were broadcasted (with the corresponding locks on the send queue), they just translate to much smaller buffers per object. Given that network traffic happens in a fully separate sending thread it would be good to understand why this impacts the overall system performance.

Possible Difficulties

No response

Prior Work

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

Trying to build this directly after cloning opencbdc-tx repository. Linking failed with:

Undefined symbols for architecture x86_64:
  "typeinfo for leveldb::Comparator", referenced from:
      typeinfo for cbdc::raft::index_comparator in libraft.a(index_comparator.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Environment

(base) ---------- 20:24:48 (duc@DucMacBook) :~/Dev2/opencbdc-tx ----------
==> clang --version
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
(base) ---------- 20:25:25 (duc@DucMacBook) :~/Dev2/opencbdc-tx ----------
==> system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro15,1
      Processor Name: 6-Core Intel Core i7
      Processor Speed: 2.6 GHz
      Number of Processors: 1
      Total Number of Cores: 6
      L2 Cache (per Core): 256 KB
      L3 Cache: 9 MB
      Hyper-Threading Technology: Enabled
      Memory: 16 GB
      System Firmware Version: 1554.140.20.0.0 (iBridge: 18.16.14759.0.1,0)
      Activation Lock Status: Disabled

To Reproduce

1. git clone --recurse-submodules https://github.com/mit-dci/opencbdc-tx
2. Change into opencbdc-tx directory and run scripts/configure.sh

(base) ---------- 20:14:06 (duc@DucMacBook) :~/Dev2/opencbdc-tx ----------
==> scripts/configure.sh
Configuring...
Warning: leveldb 1.23 is already installed and up-to-date.
To reinstall 1.23, run:
  brew reinstall leveldb
Warning: llvm@11 11.1.0_4 is already installed and up-to-date.
To reinstall 11.1.0_4, run:
  brew reinstall llvm@11
Warning: googletest 1.11.0 is already installed and up-to-date.
To reinstall 1.11.0, run:
  brew reinstall googletest
Warning: lcov 1.15 is already installed and up-to-date.
To reinstall 1.15, run:
  brew reinstall lcov
Warning: make 4.3 is already installed and up-to-date.
To reinstall 4.3, run:
  brew reinstall make
To run clang-tidy, you must add it to your path. Ex: ln -s /usr/local/opt/llvm@11/bin/clang-tidy /usr/local/bin/clang-tidy
Installing NuRaft from cache...
--2022-02-06 20:14:16--  https://raw.githubusercontent.com/llvm/llvm-project/e837ce2a32369b2e9e8e5d60270c072c7dd63827/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12311 (12K) [text/plain]
Saving to: ‘/usr/local/bin/run-clang-tidy.py.5’

run-clang-tidy.py.5               100%[=============================================================>]  12.02K  --.-KB/s    in 0s      

2022-02-06 20:14:16 (31.4 MB/s) - ‘/usr/local/bin/run-clang-tidy.py.5’ saved [12311/12311]

3. Run scripts/builld.sh

(base) ---------- 20:14:16 (duc@DucMacBook) :~/Dev2/opencbdc-tx ----------
==> scripts/build.sh    
Building...
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/duc/Dev2/opencbdc-tx/build
Consolidate compiler generated dependencies of target secp256k1_genctx
Consolidate compiler generated dependencies of target bech32
Consolidate compiler generated dependencies of target archiver
Consolidate compiler generated dependencies of target crypto
Consolidate compiler generated dependencies of target serialization
Consolidate compiler generated dependencies of target transaction
Consolidate compiler generated dependencies of target atomizer_raft
Consolidate compiler generated dependencies of target atomizer
Consolidate compiler generated dependencies of target network
Consolidate compiler generated dependencies of target common
Consolidate compiler generated dependencies of target watchtower
Consolidate compiler generated dependencies of target raft
[  1%] Building C object 3rdparty/CMakeFiles/secp256k1_genctx.dir/secp256k1/src/gen_context.c.o
[  1%] Building CXX object 3rdparty/bech32/CMakeFiles/bech32.dir/bech32.o
[  3%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_avx2.o
[  3%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer_raft.dir/atomizer_raft.o
[  3%] Building CXX object src/uhs/atomizer/archiver/CMakeFiles/archiver.dir/client.o
[  4%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/format.o
[  5%] Building CXX object src/uhs/transaction/CMakeFiles/transaction.dir/transaction.o
[  7%] Building CXX object src/util/raft/CMakeFiles/raft.dir/console_logger.o
[  7%] Building CXX object src/util/common/CMakeFiles/common.dir/buffer.o
[  8%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer.dir/atomizer.o
[  9%] Building CXX object src/util/network/CMakeFiles/network.dir/connection_manager.o
[  9%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/controller.o
[  9%] Linking C executable secp256k1_genctx
[  9%] Built target secp256k1_genctx
[ 10%] Building CXX object src/uhs/atomizer/archiver/CMakeFiles/archiver.dir/controller.o
[ 10%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_shani.o
[ 11%] Linking CXX static library libbech32.a
[ 11%] Built target bech32
[ 11%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/buffer_serializer.o
[ 11%] Building CXX object src/util/common/CMakeFiles/common.dir/hash.o
[ 12%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_sse4.o
[ 12%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer_raft.dir/controller.o
[ 12%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_sse41.o
[ 13%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/size_serializer.o
[ 13%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/stream_serializer.o
[ 13%] Building CXX object src/uhs/transaction/CMakeFiles/transaction.dir/messages.o
[ 13%] Building CXX object src/util/raft/CMakeFiles/raft.dir/state_manager.o
[ 13%] Building CXX object src/util/network/CMakeFiles/network.dir/peer.o
[ 14%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256.o
[ 15%] Building CXX object src/util/common/CMakeFiles/common.dir/hashmap.o
[ 16%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/istream_serializer.o
[ 17%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/watchtower.o
[ 18%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/siphash.o
[ 18%] Linking CXX static library libcrypto.a
[ 19%] Building CXX object src/util/common/CMakeFiles/common.dir/keys.o
[ 19%] Built target crypto
[ 20%] Building CXX object src/uhs/transaction/CMakeFiles/transaction.dir/validation.o
[ 21%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer.dir/block.o
[ 22%] Building CXX object src/util/serialization/CMakeFiles/serialization.dir/ostream_serializer.o
[ 23%] Building CXX object src/util/raft/CMakeFiles/raft.dir/log_store.o
[ 24%] Building CXX object src/util/network/CMakeFiles/network.dir/socket.o
[ 24%] Building CXX object src/util/common/CMakeFiles/common.dir/config.o
[ 24%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/status_update.o
[ 25%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer_raft.dir/state_machine.o
Consolidate compiler generated dependencies of target sentinel
[ 25%] Linking CXX static library libserialization.a
[ 26%] Building CXX object src/uhs/atomizer/sentinel/CMakeFiles/sentinel.dir/controller.o
[ 26%] Building CXX object src/util/network/CMakeFiles/network.dir/socket_selector.o
[ 26%] Built target serialization
[ 27%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/error_cache.o
[ 27%] Linking CXX static library libarchiver.a
[ 27%] Built target archiver
[ 28%] Building CXX object src/util/raft/CMakeFiles/raft.dir/node.o
[ 29%] Building CXX object src/uhs/transaction/CMakeFiles/transaction.dir/wallet.o
[ 29%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer.dir/state_machine.o
[ 30%] Building CXX object src/util/network/CMakeFiles/network.dir/tcp_listener.o
[ 30%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/block_cache.o
[ 31%] Building CXX object src/util/common/CMakeFiles/common.dir/logging.o
[ 32%] Building CXX object src/util/network/CMakeFiles/network.dir/tcp_socket.o
[ 33%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer.dir/format.o
[ 33%] Linking CXX static library libnetwork.a
[ 33%] Built target network
[ 33%] Building CXX object src/uhs/atomizer/sentinel/CMakeFiles/sentinel.dir/server.o
[ 33%] Building CXX object src/util/raft/CMakeFiles/raft.dir/serialization.o
[ 34%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/tx_error_messages.o
[ 34%] Building CXX object src/util/common/CMakeFiles/common.dir/random_source.o
Consolidate compiler generated dependencies of target sentinel_interface
[ 34%] Building CXX object src/uhs/sentinel/CMakeFiles/sentinel_interface.dir/format.o
[ 35%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/status_update_messages.o
[ 36%] Building CXX object src/util/raft/CMakeFiles/raft.dir/messages.o
[ 36%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer.dir/messages.o
[ 37%] Building CXX object src/uhs/sentinel/CMakeFiles/sentinel_interface.dir/client.o
[ 37%] Linking CXX static library libtransaction.a
[ 37%] Built target transaction
[ 37%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/messages.o
[ 37%] Building CXX object src/util/raft/CMakeFiles/raft.dir/index_comparator.o
[ 38%] Linking CXX static library libatomizer_raft.a
[ 39%] Linking CXX static library libcommon.a
[ 39%] Built target atomizer_raft
Consolidate compiler generated dependencies of target rpc
[ 39%] Built target common
[ 40%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtower.dir/client.o
[ 40%] Building CXX object src/uhs/sentinel/CMakeFiles/sentinel_interface.dir/interface.o
[ 40%] Building CXX object src/util/rpc/CMakeFiles/rpc.dir/format.o
[ 41%] Linking CXX static library libsentinel.a
[ 42%] Linking CXX static library libraft.a
[ 42%] Built target sentinel
Consolidate compiler generated dependencies of target shard
[ 43%] Building CXX object src/uhs/atomizer/shard/CMakeFiles/shard.dir/shard.o
[ 43%] Built target raft
Consolidate compiler generated dependencies of target client
Consolidate compiler generated dependencies of target coordinator
[ 43%] Building CXX object src/uhs/client/CMakeFiles/client.dir/atomizer_client.o
[ 43%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/format.o
[ 44%] Building CXX object src/uhs/client/CMakeFiles/client.dir/client.o
[ 45%] Linking CXX static library libatomizer.a
[ 45%] Built target atomizer
Consolidate compiler generated dependencies of target locking_shard
[ 46%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/client.o
[ 46%] Building CXX object src/uhs/atomizer/shard/CMakeFiles/shard.dir/controller.o
Consolidate compiler generated dependencies of target sentinel_2pc
[ 46%] Building CXX object src/uhs/twophase/sentinel_2pc/CMakeFiles/sentinel_2pc.dir/controller.o
[ 46%] Building CXX object src/uhs/client/CMakeFiles/client.dir/twophase_client.o
[ 47%] Linking CXX static library librpc.a
[ 47%] Built target rpc
[ 48%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/state_machine.o
[ 48%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/controller.o
[ 49%] Linking CXX static library libsentinel_interface.a
[ 49%] Built target sentinel_interface
[ 50%] Building CXX object src/uhs/twophase/sentinel_2pc/CMakeFiles/sentinel_2pc.dir/server.o
[ 51%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/locking_shard.o
[ 51%] Linking CXX static library libwatchtower.a
[ 51%] Built target watchtower
Consolidate compiler generated dependencies of target util
[ 52%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/client.o
[ 53%] Building CXX object tests/CMakeFiles/util.dir/util.o
[ 54%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/interface.o
[ 54%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/distributed_tx.o
[ 55%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/controller.o
[ 56%] Linking CXX static library libshard.a
[ 56%] Built target shard
[ 56%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/format.o
[ 56%] Building CXX object src/uhs/twophase/coordinator/CMakeFiles/coordinator.dir/server.o
[ 57%] Built target secp256k1_genctx_run
[ 58%] Linking CXX static library libclient.a
[ 59%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/messages.o
[ 59%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/state_machine.o
[ 60%] Linking CXX static library libsentinel_2pc.a
[ 60%] Built target client
[ 61%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/status_client.o
[ 61%] Built target sentinel_2pc
Consolidate compiler generated dependencies of target secp256k1
[ 62%] Building C object 3rdparty/CMakeFiles/secp256k1.dir/secp256k1/src/secp256k1.c.o
[ 63%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking_shard.dir/status_server.o
[ 63%] Linking C static library libsecp256k1.a
[ 63%] Built target secp256k1
Consolidate compiler generated dependencies of target archiverd
Consolidate compiler generated dependencies of target atomizer-raftd
Consolidate compiler generated dependencies of target sentineld
[ 65%] Building CXX object src/uhs/atomizer/archiver/CMakeFiles/archiverd.dir/archiverd.o
[ 65%] Building CXX object src/uhs/atomizer/atomizer/CMakeFiles/atomizer-raftd.dir/atomizer-raftd.o
[ 65%] Building CXX object src/uhs/atomizer/sentinel/CMakeFiles/sentineld.dir/sentineld.o
Consolidate compiler generated dependencies of target shardd
[ 66%] Building CXX object src/uhs/atomizer/shard/CMakeFiles/shardd.dir/shardd.o
[ 67%] Linking CXX static library libutil.a
Consolidate compiler generated dependencies of target watchtowerd
[ 67%] Built target util
[ 68%] Building CXX object src/uhs/atomizer/watchtower/CMakeFiles/watchtowerd.dir/watchtowerd.o
Consolidate compiler generated dependencies of target atomizer-cli-watchtower
[ 68%] Building CXX object tools/bench/CMakeFiles/atomizer-cli-watchtower.dir/atomizer-cli-watchtower.o
Consolidate compiler generated dependencies of target shard-seeder
[ 69%] Building CXX object tools/shard-seeder/CMakeFiles/shard-seeder.dir/shard-seeder.o
[ 69%] Linking CXX executable archiverd
[ 70%] Linking CXX executable sentineld
[ 70%] Linking CXX static library liblocking_shard.a
[ 70%] Built target sentineld
[ 70%] Built target locking_shard
[ 70%] Built target archiverd
Consolidate compiler generated dependencies of target locking-shardd
[ 71%] Building CXX object src/uhs/twophase/locking_shard/CMakeFiles/locking-shardd.dir/locking_shardd.o
[ 72%] Linking CXX executable watchtowerd
[ 72%] Linking CXX executable shardd
[ 72%] Linking CXX executable atomizer-raftd
[ 72%] Linking CXX executable shard-seeder
Undefined symbols for architecture x86_64:
  "typeinfo for leveldb::Comparator", referenced from:
      typeinfo for cbdc::raft::index_comparator in libraft.a(index_comparator.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/uhs/atomizer/atomizer/atomizer-raftd] Error 1
make[1]: *** [src/uhs/atomizer/atomizer/CMakeFiles/atomizer-raftd.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 72%] Built target shardd
[ 72%] Built target watchtowerd
[ 72%] Built target shard-seeder
[ 73%] Linking CXX static library libcoordinator.a
[ 73%] Built target coordinator
[ 73%] Linking CXX executable locking-shardd
Undefined symbols for architecture x86_64:
  "typeinfo for leveldb::Comparator", referenced from:
      typeinfo for cbdc::raft::index_comparator in libraft.a(index_comparator.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/uhs/twophase/locking_shard/locking-shardd] Error 1
make[1]: *** [src/uhs/twophase/locking_shard/CMakeFiles/locking-shardd.dir/all] Error 2
[ 74%] Linking CXX executable atomizer-cli-watchtower
[ 74%] Built target atomizer-cli-watchtower
make: *** [all] Error 2
(base) ---------- 20:14:49 (duc@DucMacBook) :~/Dev2/opencbdc-tx ----------
==> 

Code of Conduct

• I agree to follow this project's Code of Conduct

There are a great deal of tests that should be added, and many additional branches that should be covered. To that end, this issue is meant to be a centralized place to keep track of tests that have been identified as good opportunities for improvement. We will try to keep the task list below up-to-date; so if you think another item should be added, feel free to comment it below and we will add to the list as makes sense!

Note: task lists allow for each item to easily be converted into a stand-alone issue; If a particular test/improvement merits deeper discussion as someone seeks to implement it, we can always make that change!

• Add tests to check the thread-safety of any/all classes which are expected to be thread-safe
• #111
• #117
• Add unit tests for cbdc::raft::rpc::server
• #133
• #108
• Add replication test to specifically check that Atomizers handle node failure gracefully
• #2
• Add integration test to check that Shards always attest at the correct block height
• Add integration tests for Locking Shards
• Add integration tests for Coordinators
• Add integration tests for 2PC Sentinels
• Add integration tests for Atomizer Sentinels
• #136
• Add support for OS-assigned ephemeral port numbers and leverage them in-tests
• #11
• Add github action to check that new PRs don't decrease test coverage
• #127

Question

What's the logic behind minting NFTs on a cbdc?

Honestly just looking to dynamically recordkeep at a high throughput.

Benefit

Potential business logic value

Proposed Solution

I'll keep reading the documentation and relay any progress here (RTFM)

Possible Difficulties

I have yet to quite understand whether minting is chain specific.

Prior Work

I read alot.

Code of Conduct

• I agree to follow this project's Code of Conduct

Question

It would be useful to explore the usage of blind signatures as a means of adding a privacy element to the system. As articulated by David Chaum, here, and then later implemented in DigiCash and GNU Taler, and most recently presented as a hypothetical CBDC implementation with the Swiss National Bank.

At the surface, it seems a simpel alternative over zero-knowledge proofs, shoudl fit the current proposed architecture, and lend itself to be quite usable in many network models.

Benefit

Blind signatures are a relatively straightforward usage of cryptographic signatures to add privacy. There is a history of usage and previous solutions to lend knowledge and expertise for implementations (Digicash, GNU Taler). Finally, the original usage of blind signatures for privacy in digital currency seems a good fit for the existing centrally operated architecture.

Proposed Solution

So far, this proposal has only identified a potential fit for the blind signature scheme. Next-step work would require a deeper analysis of previous blind signature implementations, integration with the existing codebase, and finally develop an initial understanding of the trade-offs and benefits versus other design choices.

Possible Difficulties

The usage of a UXTO model like UHS was not present in the previously mentioned implementations.

Prior Work

Original description of blind signature proposal, https://sceweb.sce.uhcl.edu/yang/teaching/csci5234WebSecurityFall2011/Chaum-blind-signatures.PDF

Open-source implementation of blind-signature wallet system, https://taler.net/en/index.html

SNB and David Chaum CBDC paper, https://www.snb.ch/en/mmr/papers/id/working_paper_2021_03

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

main @ https://github.com/mit-dci/opencbdc-tx.git

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

While running "Setup test wallets and test them":
./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 10 5

[2022-02-09 19:08:44.231] [WARN ] Existing wallet file not found
[2022-02-09 19:08:44.232] [WARN ] Existing client file not found

following, we see an error:
"""
./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat info
Balance: $0.50, UTXOs: 10, pending TXs: 0
root@88cc0c05fa6d:/opt/tx-processor# Balance: $0.50, UTXOs: 10, pending TXs: 0
bash: Balance:: command not found
root@88cc0c05fa6d:/opt/tx-processor# ./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat sync
[2022-02-09 19:10:34.623] [ERROR] Failed to connect to any atomizers
terminate called after throwing an instance of 'std::system_error'
what(): Invalid argument
"""

Code of Conduct

• I agree to follow this project's Code of Conduct

Hi team,
I want to build a system written in Java to integrate with open cbdc-tx. Is there any way to execute these operations (mint, transfer, redeem) using Java?
Thank you.

https://clang.llvm.org/docs/ThreadSafetyAnalysis.html

Makes debugging race-conditions and other potential bugs in highly-threaded code much more tractable!

Affected Branch

Scanning dependencies of target crypto
[ 4%] Building CXX object 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_avx2.o
c++: error: unrecognized command line option '-mavx'
c++: error: unrecognized command line option '-mavx2'
make[2]: *** [3rdparty/crypto/CMakeFiles/crypto.dir/build.make:63: 3rdparty/crypto/CMakeFiles/crypto.dir/sha256_avx2.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:771: 3rdparty/crypto/CMakeFiles/crypto.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
The command '/bin/sh -c mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} .. && make' returned a non-zero code: 2
ERROR: Service 'shard0' failed to build : Build failed

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

In order to reproduce the issue, follow these steps:

Having trouble with the build got the error above.

Code of Conduct

• I agree to follow this project's Code of Conduct

At the moment, the coordinator state machine stubs out not-yet-implemented functions for snapshots. These should be replaced by functional implementations.

Abstract

Error with the docker build: Unable to locally verify the issuer's authority.

Description

Error with the docker build.

bash-3.2# docker build . -t opencbdc-tx

"""
--2022-02-03 22:41:59-- https://github.com/google/leveldb/archive/1.22.tar.gz
Resolving github.com (github.com)... [IP]
Connecting to github.com (github.com)|[IP]|:443... connected.
ERROR: cannot verify github.com's certificate, issued by '....':
Unable to locally verify the issuer's authority.
To connect to github.com insecurely, use `--no-check-certificate'.
"""

After adding --no-check-certificate in 2 lines of Dockerfile, I hit another error

"""

• . manifest.sh
  ++ ASIO_RELEASE=asio-1-12-0
  ++ pwd
• PROJECT_DIR=/NuRaft-1.2.0
• cd /NuRaft-1.2.0
• '[' '!' -d asio ']'
• git clone https://github.com/chriskohlhoff/asio.git ./asio
  Cloning into './asio'...
  fatal: unable to access 'https://github.com/chriskohlhoff/asio.git/': server certificate verification failed. CAfile: none CRLfile: none
  """

I also tried

"""
RUN apt-get install ca-certificates # added to Dockerfile
"""
which gives
"""
E: Package 'ca-certificates' has no installation candidate
"""

Code of Conduct

• I agree to follow this project's Code of Conduct

We've chosen a UTXO-based model, but we should track all the pluses/minuses we end up finding for each here.

• Coin selection, aggregation (-UTXOs)
• Have to "discover" my money; hard to know what my UTXOs are (-UTXOs)
• Having to make change (-UTXOs)

Currently archiverd is not replicated. Use Raft or some other protocol to make the archiver fault tolerant.

Often, a cryptographic system wants to make use of multiple independent random oracles, and when this random oracle is instantiated with the same hash function the security argument breaks down. E.g. Separate Your Domains: NIST PQC KEMs, Oracle Cloning and Read-Only Indifferentiability breaks multiple NIST PQC candidates which reuse the same H(...) for multiple purposes.

I am not aware of any exploitable instances in our codebase, but domain separation is a good practice, especially as the system grows more complex. In general, domain separation is more relevant for computations involving secrets (out of scope for us) or withheld data (definitely in scope for us, as the system maintains minimum data). For a Bitcoin-specific exploitable instance related to the latter one can remember the Merkle tree serialization confusion (this would have been prevented if different layers of the Merkle tree had different random oracles or, equivalently, one committed to the depth in the each hash computation). Domain separation also something people looking at our code will notice and ask about.

@theuni suggested looking at the Schnorr Signature BIP 340, which proposes calculating hash H_tag(data) as H(H(tag) || H(tag) || data). With this choice and when H = SHA256 one can optimize the first block computation.

Question

Can strong Layer 1 privacy solutions, currently deployed in cryptocurrencies, be scaled to a level of a major payment system or a CBDC? What are trade-offs and implications for payment workflows, auditability of supply, and regulatory compliance?

Benefit

OpenCBDC is designed to store minimal information at the transaction processor core: the main data structure (UHS) consists of opaque UTXO hashes.

However, this is not enough for policymakers who desire strong privacy: the transactions submitted to OpenCDBC sentinels reveal sender and recipient addresses, as well as the payment amount. In a case of compromised sentinels (e.g., by foreign actors), the system could, in principle, retain the entire transaction graph. Similarly, compromised atomizer or 2PC logs might give limited insight into the transaction graph.

The best way to safeguard information is to never have it in the first place. This proposal describes how to explore strong cryptographic techniques, including, encryption and zero-knowledge proofs, to make sure that system operators can be confident as to never touch sensitive financial information in the clear.

Proposed Solution

The first step of our proposal is to explore the commonalities of successful L1 privacy coin designs, e.g. CryptoNote, Zerocash, Zexe, and others (implemented in Monero, Zcash, Aleo, and others), and their current and proposed extensions.

At the transaction processing level these designs share remarkable commonalities. For example, all of them maintain a list of all transaction outputs ever created. To spend a transaction output, one reveals a value uniquely derived from an output, but unlinkable by an external observer; these values are called nullifiers (also called key images in CryptoNote/Monero or serial numbers in the seminal work of Sander and Ta-Shma, and in Zerocash and Zexe. NB: "serial numbers" in OpenCBDC serve a different purpose, prompting us to adopt the nullifier nomenclature in this description).

To prove that the nullifier was derived from a valid, existing coin, the sender can, for example:

• pick a set of other prior outputs (called decoys) and create a linkable ring signature involving the sender’s real output plus all the decoys; or
• compute a Merkle tree of all preceding outputs, and prove that the coin the sender is spending is in this Merkle tree, and its nullifier is consistent with the one sender is revealing.

There are high-quality open-source implementations for the cryptographic components described above. What we propose exploring in the first phase of this work is to see the feasibility to integrate these with OpenCBDC.

At a high level, OpenCBDC uses UTXO hash set (UHS) and the core transaction processing is represented by a swap primitive (see Section 3.3 Transaction format and execution in the whitepaper). The inputs to swap are two lists of hashes: one for existence checks and removal (called input hashes), and one for insertion (called output hashes). To execute a swap, the system atomically checks that all input hashes are present. If an input hash is missing, swap aborts. Otherwise, it obtains an updated UHS by erasing all input hashes and inserting all output hashes.

To support efficient processing of private transactions, we would need to build a system that exposes a different core primitive; let’s call it append (better naming suggestions welcome!). Now, a system maintains a set N of nullifiers, and a list L of outputs. The inputs to append are also a set N’ of nullifiers (for non-existence checks and insertion in the nullifier set), and a list L’ of outputs (for insertion of the global list of outputs). To execute an append, the system atomically checks that none of N’ nullifiers are in N. If a nullifier is already present, append aborts. Otherwise, it updates the system state by adding nullifiers in N’ to N and appending hashes in L’ to L.

The required modifications at a sentinel level depend on the chosen privacy techniques:

• To support ring signatures, a sentinel would need random access to all prior decoy outputs.
• To support Merkle tree-based approaches, the system would periodically compute a Merkle tree over all outputs in L and publish it. A sentinel would need random access to all these prior Merkle tree roots.

Note that for the latter scenario, Merkle trees can be updated lazily (not atomically), and there can be a Merkle tree per shard. That is, for the higher level protocols it suffices to prove that a commitment is included in a shard-specific Merkle tree (potentially hiding the tree using tree-of-trees on top); not necessarily one globally maintained MT.

(We remark that, In principle, the non-existence checks can be implemented using the existing swap abstraction in a black-box way by storing unused nullifier intervals in UHS IDs. Then a transaction would consume such interval and produce one or two new intervals for the split. However, this adds application-layer complexity and a non-black-box implementation might be preferred.)

The above description is aimed just to start a discussion about Layer 1 privacy which is a complex topic, and involves many parts not mentioned here. For example, system would need different auxiliary services (similar to how OpenCBDC has watchtowers) to help users create their proofs (as users do not have ability to run a full node), it would require making decisions related to payment discovery (see issue #28), auditability, compliance, and so on.

Possible Difficulties

Successful execution of this project has a significant policy component, and will require clarity from policymakers (esp., related to compliance goals such as deterring illicit finance). This makes the project’s scope open-ended.

Prior Work

• Auditable, Anonymous Electronic Cash (Sander, Ta–Shma; 1999)
• CryptoNote v2.0 (van Saberhagen; 2013)
• Zerocash: Decentralized Anonymous Payments from Bitcoin (Ben-Sasson, Chiesa, Garman, Green, Miers, Tromer, Virza; 2014)
• Zexe: Enabling Decentralized Private Computation (Bowe, Chiesa, Green, Miers, Mishra, Wu; 2018)
• Scalable Privacy (Hopwood; 2019) (slides)

Code of Conduct

• I agree to follow this project's Code of Conduct

We've seen several instances of people trying to build the code on slightly (or wildly) different platforms from the one we “officially” support (namely, x64 Linux), and it resulting in similar build errors (cf. #35)

In particular, our build system currently hard-codes some flags that we want to set for performance, but aren't available on all platforms. It would be ideal if we could teach our CMake files to dynamically define performance-related flags based on the target architecture to fix these problems more systemically (than offering specific suggestions to people to fix the build for their target). This should also just generally make the code more portable!

Requested by @theuni -- consider ways to replicate the atomizer without using raft.

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

This issue was initially found and reported by @toddfratello, and has been confirmed using the procedure below.

To reproduce in docker, we apply the following patch to enable use of nc (netcat) inside the containers (this is only for ease-of-testing, the bug is present without this patch):

diff --git a/Dockerfile b/Dockerfile
index 6fe2b54..41c06ff 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,6 +12,7 @@ RUN apt update && \
       libgtest-dev \
       libgmock-dev \
       net-tools \
+      netcat \
       git

 # Args

The above will require rebuilding the docker image:

# docker build . -t opencbdc-tx

After rebuilding the docker image, the following can reliably reproduce the issue

1. Run the 2PC architecture:

   # docker-compose --file docker-compose-2pc.yml up

2. Run a separate container, connected to the 2PC network:

   # docker run --network 2pc-network -ti opencbdc-tx /bin/bash

3. In the separate container, send the coordinator a stream of random bytes:

   # cat /dev/urandom | nc 172.25.0.3 7777

   NB: the IP address used above may be dependent on your docker configuration; but you can find the specific IP address of your coordinator in your docker setup using docker network inspect 2pc-network.
4. Note that the coordinator errors out and restarts.

In particular, the issue appears to be caused by this line:

In the PoC above, pkt_sz is very large, and attempting to allocate so much space results in the Coordinator dying with the following error:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Code of Conduct

• I agree to follow this project's Code of Conduct

Right now, each shard includes the entire compact_tx in the tx_notify_request message to the atomizer when it attests to a transaction's inputs.

If a transaction spans multiple shards, the compact_tx is getting sent to the atomizer n_shards times.

Instead, the shard could send a smaller (compacter?) transaction which just includes the inputs it is attesting to. Perhaps the lowest index shard could be responsible for sending the transaction's outputs.

Affected Branch

The hyperlink for the Technical Reference in getting-started.md goes to a 404 site. Not sure if its an issue with the host site or if the link is incorrect.

* The [Technical Reference](https://mit-dci.github.io/opencbdc-tx/) includes documentation for the code itself and how it works

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

In order to reproduce the issue, follow these steps:

1. Click hyperlink text "Technical Reference" in getting-started.md

Code of Conduct

• I agree to follow this project's Code of Conduct

Right now, a full_tx is an SoA (“struct of arrays”). This means, at minimum, we include 8 extra bytes in its representation (the count of witnesses is superfluous because all valid transactions must have an identical number of witnesses to inputs). Moreover, this likely poses an optimization opportunity (because when we are accessing a particular witness, we are also likely to be accessing its associated input; as a result, greater cache locality can probably be achieved by storing them adjacent to one another).

To implement this, we would add a new structure that pairs a witness with its associated input, and update full_tx to have a vector of these structs rather than grouping together the two vectors.

Note: this change might be rendered immaterial should we implement #6

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

The 2pc locking shard includes leveldb as a dependency. https://github.com/mit-dci/opencbdc-tx/blob/trunk/src/uhs/twophase/locking_shard/locking_shard.hpp#L20. But it doesn't look like leveldb is actually being used.

Removing the include from locking_shard.hpp builds without errors. However removing leveldb from CMakeLists.txt: https://github.com/mit-dci/opencbdc-tx/blob/trunk/src/uhs/twophase/locking_shard/CMakeLists.txt#L23 results in build errors. This has been confirmed by @HalosGhost

Code of Conduct

• I agree to follow this project's Code of Conduct

This check was removed from automated-linting when we upgraded clang-tidy to version 13; it would require significant, non-critical-path refactoring.

However, we should work to get the code conformant and reenable the check.

Since we're using hash functions for various identifiers and relying on collision-resistance of these identifiers, one optimization is to have a keyed hash function where only the server side knows the key.

The advantage of this is that even with a 16 byte truncated hash, users would not be able to cause identifier collisions, as they never get to see the TXIDs / UTXOids to try to collide them.

The downside is that users can't compute identifiers themselves. So they can put together a transaction and sign it, but not compute the TXID themselves. Or they could have a user-facing TXID / UTXOid hash function that they can compute, but the internal usage is keyed.

The internal usage could also be a keyed hash of the public hash. So for example, users compute their TXID as sha256(transaction) and internally we use ITXID = hmac( cbdc_secret, TXID). We can then truncate the hmac output to 16 bytes without worrying about collisions.

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

There are several small issues with both scripts/configure.sh and scripts/build.sh that prevents building on OSX. Here are few I've run across so far:

• brew installs leveldb but the header files are not available to the build. There's also the issue of using different versions of leveldb between platforms: Linux build uses version 1.2.2. brew uses the latest which is 1.2.3
• configure.sh uses wget in a few places but wget is not installed by default on OSX. curl is available on OSX
• build.sh hardcodes a path to gmake which is different from the default location from brew
• copying nuraft headers, etc... to /use/local/ requires sudo
• git submodules are not pulled in either script.

Note: I'm working on a fix for all this and will submit a pull request once I've worked through all the issues.

Code of Conduct

• I agree to follow this project's Code of Conduct

Right now, we have an operator>> that can deserialize std::variants whose alternatives are all default-constructible, and we provide get_variant() to deserialize std::variants whose alternatives are all not default-constructible (leveraging a special constructor instead).

It should not be too difficult to extend get_variant() to also support types that are default-constructible (so that it can support std::variants that have some alternatives that are default-constructible and some that are not).

Following some previous work, we can test spinning up multiple shards at once; now we should make sure they behave well through replication and partitioning.

The config crafted for the spin-up tests already lays out ranges for the shards such that among the five shards, no single transaction should ever live in-range of more than three shards, and the shards at both ends of the range cover areas that no other shard covers (so no other config changes should be necessary to test either replication or partitioning). Below are the minimal scenarios we should plan to test:

• multiple sentinels sending to the right set of shards
• shards should only attest for inputs in their range
• shards should only update their output set for those that are in their range
• watchtower error messages emitted by shards (when shards receive transactions that include inputs which are in-range but don't exist)

Also add a unit test for this case, and re-enable the disabled archiver unit test marked with a TODO related to this failure.

here, users are incorrectly told one needs a root shell for Docker. This is not true. The Docker installation instructions themselves detail specifically how to avoid this, and if you are using native mac/windows apps you probably don't even have to do that.

I suggest either:

• no instructions are given, and users are trusted to know how to use Docker, or how to look that up themselves
• if such hand-holding is desired/necessary, we encourage best practices and point users towards the guide above rather than encourage unnecessary use of root permissions

Background

Right now, we have two architectures (2PC and Atomizer), so when a new feature gets proposed, there is a high-probability that implementation will require touching two separate locations in the code.
As we explore more architectures, or solutions which involve changes to the two we have, this problem is likely going to be more pronounced.
It results in the following draw-backs:

• The possibility for architectures to get out-of-sync in feature-support.
  • Accepting this leads to an extra documentation burden (documenting the matrix of which architecture supports which features).
  • Avoiding this leads to an ever-increasing implementation burden (a new feature will need to intentionally interoperate will all previously-merged features).
• Testing with a new feature (say, B) but without an old one (say, A) is tedious.
  • Accepting this would require the following overhead: checkout a new branch, reset to a commit before A, cherry-pick the relevant commits for B, and fix conflicts.
  • Avoiding this leads to treating any sufficiently-fundamental change to an architecture being treated as a new architecture.
• The mental burden necessary to get involved grows dramatically with every new feature (as even knowing which architecture you want to use becomes much less intuitive).
  • Accepting this leads to fewer contributions.
  • (Even partially) avoiding this leads to dramatically increased documentation burden (trying to explain not just which architectures support which features, but why and which you might want to use for a given experiment).
• Supplemental tooling (e.g., docker compose configurations) is subject to more and more significant duplication/complexity/churn.
  • Accepting this may mean that we end up with a very large library of docker compose configurations (one for each of the now-many architectures), which also implies frequent additions of new tooling solutions for each new architecture/feature.
  • Avoiding this either results in a much more complex setup for the supporting tooling such that it can handle the complexity above, or simply not supporting some architectures with tooling as well as others (additional maintenance or documentation burden, or both)

In short, we know there are a lot more architectures and features worth exploring; to ensure project sustainability, we need to develop patterns and conventions for adding new features in such a way that we can minimize the potential combinatorial expansion shown above.

A Solution

Feature flags offer a potential avenue to refactor the code such that new features are gated behind a flag to change the system's behavior at run-time.
This would allow, for example, many different implementations of a sentinel to coexist and for which implementation gets used to be selected at run-time (rather than build- or deploy-time).

Taking this possibility to its logical conclusion, an architecture becomes a particular set of components being run with a particular set of feature flags (e.g., 2PC and Atomizer could be partly merged and we could have a flat component list).

Pros

Assuming we could come up with a perfect feature flags system, it would enable the following benefits:

• Changes to current architectures (even deeply fundamental ones) only require touching the specific, affected locations.
• New architectures can be added by adding necessary flag-gated features to current components and adding any new components.
• Testing with a combination of features that hasn't been considered before becomes much more tractable (just set the feature flags and run the needed components).
• Contributors can care only about which other features a new feature/architecture needs to interact with (rather than with all of them).
• New contributors see the flat list of components and a small document detailing known configurations which result in particular outcomes is much more approachable.
• Supplemental tooling (as long as it can be taught to interpret/leverage the feature flags) can be much more ever-green (so long as the feature-flags system is capable of containing a new change, tooling may not need to change at all).
• May increase potential velocity of new contributions by minimizing additional technical debt associated with adding new features/components.

Cons

Still assuming that the system we adopt/implement is ideal for our use-case, there are some draw-backs:

• It becomes possible to run the system in multiple configurations that won't work (potentially, more options than ones that would work even).
  • E.g., not running an archiver if you want to run a variant of the Atomizer architecture will result in several failure modes.
  • This is specifically because feature flag solutions make most sense at the "how the program works" level (not the "what program gets run" level).
    It is possible we could additionally add a notion of dependencies between components (perhaps influenced by feature flags) to mitigate this, but such a solution isn't clear.
• Adapting to such a system requires significant changes to our current layout and tooling.
• The clarity and meaning of something being "an architecture" is muddied (this leads to benefits mentioned above, but does make architecture discussions/research more nuanced and as a result more difficult).
• Some changes might be too fundamental for the feature flags system to handle.
  • The obvious solution would be to "fork" the changed component (note: not a github fork, but rather a new component initially copied from the old one), but drawing that line becomes a nuanced decision to make.

Practical Issues

Beyond the costs and benefits mentioned above, there's a much more practical problem: feature flags aren't well standardized (because they operate inside the application-layer's logic, implementations tend to be very domain-specific).
Additionally, using feature-flags may result in control flow in hot-path code (significant performance degradation).

Pre-made

There are off-the-shelf solutions we could try to adopt, but they each have their draw-backs; below is a list of known solutions we could adopt and known draw-backs (more will be added as they are suggested and evaluated):

• Backtrace's dynamic_flag: it's x86_64-specific (leverages inline assembly), and only supports boolean flags
• A great many command-line option parsing implementations: unifies settings (knobs to turn to change specific bits of behavior) and arguments (e.g., the client's newaddress command) with feature-flags (potentially signifcant behavior changes)
• Some work is underway to create a unified specification for feature flags in OpenFeature; it's still early days, but this may become the ideal option in the future.

Custom-built

This is the most sensible option in a lot of ways (we build it for exactly what we need and want to support), but has its own issues.
Namely, we actually need to design and implement a feature flags system (which, though enabling future work, is orthogonal to our research purpose).

Conclusion

Ultimately, some version of organizing the code to handle the combinatorial expansion of features/architectures is probably necessary, but I do not know if feature-flags is really the correct solution (another might be to maintain separate branches for each architecture, though that also has obvious drawbacks).
Comments, questions, clarifications, and suggestions are all welcome!

Question

What amount of smart contract support is possible with the UTXO model and CBDC architecture?

Benefit

A CBDC with smart contract support could be useful to both policy makers and end users:

• Policy makers would have the ability to automate policy through code.
• End users could pay taxes or establish things such as re-occurring payments. 
• Business could use the functionality to automate payments or escrow funds. 

Practically speaking, while this all sounds valuable, it’s important to find the right balance between the desired smart contract feature set and the impact it could have on the overall architecture.

Proposed Solution

The UTXO approach has many valuable features: scalable, simpler, and deterministic. Many of these benefits come from the fact that it doesn’t require transactions to operate on global state.  But the lack of state can make it challenging to provide smart contract support beyond the simple script support in Bitcoin.
 
Potential research:

1. Investigate cryptocurrency approaches and how they may be used in a CBDC. For example, platforms such as Cardano, Bitcoin Cash, and Ergo are evolving smart contract support for the UTXO model via efforts such as extended UTXO (eUTXO) and multi-stage UTXO. 
2. Explore the use of Web Assembly as a CBDC smart contract environment. Blockchain platforms such a Near, Substrate/Polkdadot, and CosmWasm use Web Assembly (WASM) as a smart contract runtime environment.  Wasm is an open standard for a portable binary and runtime with the goals of being safe and deterministic

Possible Difficulties

Adding smart contract support will affect (at a minimum) the transaction format and transaction processor.  This could impact performance as well as further complicate a privacy solution. 

Prior Work

• https://sundaeswap-finance.medium.com/concurrency-state-cardano-c160f8c07575
• https://ergoplatform.org/en/blog/2021-07-23-ergo-utxo-model-the-evolution-from-bitcoin-to-ergo/
• https://reference.cash/protocol/blockchain/script
• https://webassembly.org/
• https://docs.substrate.io/v3/advanced/executor/
• https://docs.near.org/docs/develop/contracts/overview
• https://cosmwasm.com/#howitworks

Code of Conduct

• I agree to follow this project's Code of Conduct

Affected Branch

trunk

Basic Diagnostics

• I've pulled the latest changes on the affected branch and the issue is still present.

• The issue is reproducible in docker

Description

In order to reproduce the issue, follow these steps:

1. I build opencbdc project and excute on ubuntu lts 20.04 on real machine (not in docker)
2. start the sentineld,coordinatord,locking-shardd on machine a for one sharding
3. then i modify config ,and start sentineld,coordinatord,locking-shardd on two machine（machine a ,machine b） for two sharding ,for machine a i have clear file shad0*.data shad*_log,coordinator*.data,coordinator*log,
4. but start locking_shard on machine a failed ,report error" [FATAL] Unable to find dtx data for apply XXXX)

how we can start locking_shard?Are there any other cache files that need to be cleaned?

Code of Conduct

• I agree to follow this project's Code of Conduct