Context

We want to have CI set up for the new cardano-wallet in order to have a means to automatically build the code and run all our tests against the build. At this moment we decided to go with Travis, so we want to get the .travis.yaml from the previous cardano-wallet review it and remove all unnecessary dependencies to cardano-sl from it. Instead, we want to depend on cardano-http-bridge as a gate for receiving blocks from the mainnet/testnet.

Decision

In order to use this repo with cardano-http-bridge we need to add instructions for building and caching Rust http-bridge to .travis.yml

Development Plan

• I intend to review the build instruction locally
• I intend to review travis documentation about Rust to see if there's anything good to know about it
• I intend to add necessary build instruction of the http bridge in the build steps of the CI, trying to make it a job running in parallel from our compilation
• I intend to make sure that the cardano-http-bridge is within the available PATH such that we can refer to it in our various scripts.

PR

QA

• cf PR comment

Context

In Shelley, addresses are encoded in bech32. A reference implementation (in Haskell) for bech32 encoding available here.

Decision

• Evaluate the quality of this reference implementation
• Extract it as a package and re-locate it in our lib/
• Add extra tests, if necessary
• Adjust the code to comply to our coding standards
• Adjust the reference implementation API to be eas(ier)y-to-use (for instance, having a decode :: Text -> Maybe Text or similar could be quite handy).

Acceptance Criteria

1. We must support bech32 encoding & decoding
2. We should provide a comprehensible and easy-to-use API for this module

Development Plan

PR

QA

• The reference implementation has been ported and adjusted to remove the bits that were bitcoin-specifics; we end up with a tiny API that is visible on haddock

• The code is fully tested here:

  • We've ported existing tests from the reference implementation see Codec.Binary.Bech32Spec
  • We've also added roundtrip tests for most of the conversions functions, see Codec.Binary.Bech32Spec

• There's an extra PR which outlines some work to perform error-detection (bech32 allows for identifying with a rather good precision where an error is located within the address string). This is still experimental and unpolished so, might be done in another ticket for it is a bit out of scope for this particular one.

Context

Coin selection happens in order to choose UTxO from the available ones in a wallet to cover (when possible) for the cost of a transaction. More details in About Coin Selection

In the context of this new wallet backend, we will need a subset of the coin selection previously developed in the legacy code base: see Kernel/CoinSelection.

The first module:

• Kernel/CoinSelection/Generic provides a generic DSL to perform CoinSelection on a ledger relying on UTxO accounting.

• Kernel/CoinSelection/FromGeneric provides an implementation of the aforementioned DSL for Cardano.

• Kernel/CoinSelection/Generic/Fee contains logic to perform fee adjustment on a selection yield by the above modules.

• Kernel/CoinSelection/Generic/Grouped is now irrelevant as we won't be considering grouping

• Kernel/CoinSelection/Generic/LargestFirst provides a simple heuristic from selecting coins from the largest first. This is the fallback heuristic in case where the random selection fails to yield a satisfactory solution. If the largest first doesn't yield any result, we know that there are not enough funds to cover the transaction cost and, a failure should be reported to users.

• Kernel/CoinSelection/Generic/Random contains logic for performing a random selection while improving the selection to minimize the discrepancy between change outputs and normal outputs.

Our final implementation only truly use a subset of the available features from the coin selection "engine", so, it is a legitimate question to ask whether we should port everything in there and preserve the existing format. The generic abstraction makes it possible to have a code that is fairly agnostic to any specificity of a particular coin or wallet. This code has also gone through several round of testing and QA, and despite having some known limitations, is expected to work fairly okay now. On the other hand, it adds several layers of indirection that are probably not needed since we do not intend to support different coins and policy now (and probably not in a near future). Not mentioning that, most of the property and unit testing happens on a concrete implementation using legacy types. So, there's not much from the testing we can re-use.

Decision

We'll port relevant part of the coin selection algorithm, dropping the generic approach and fallback to a specialized implementation using our primitive types. Doing so, we'll also NOT port:

• Generic DSL
• PrivacyModeOff approach from the random selection
• Grouping policy
• Receiver pay fee

Instead, we'll focus on providing a concrete implementation that cover our current and upcoming needs, testing as we go and preserving the algorithms as they currently exists in the generic coin selection.

The legacy coin selection implementation is also known to be quite inaccurate in the way it reports error. So, we do want to be extra careful when dealing with errors and make sure we make the implementation very honest and helpful about errors.

In order to simplify the implementation work for this particular bit, we will also push back on the exact fee calculation for transaction and leave the implementation flexible regarding fees, using hard-coded constant for now (and for this, we may consider an interface closely resembling to FeeOptions).

Acceptance Criteria

• We must be able to perform coin selection from available UTxOs, and yield a result in the form of:

  • a list of inputs
  • a list of outputs
  • a list of change coins (later mapped to change addresses)

• Coin selection must preserve privacy and generate, as much as possible, outputs and change addresses that are roughly of the same size (this can be achieved by porting the algorithms used in the legacy coin selection).

• Coin selection should make the sender pay fees, removing fee from its original change

• We should preserve the separation of concerns between coin selection and fee adjustments.

• We may port the various integration test scenarios about coin selections that exist on the legacy code base in the form of unit tests, and control that selection are yielded correctly

• Coin selection shouldn't include any form of grouping at this stage

• Explicit fee calculation and estimation from a transaction may be mocked with dummy values at this stage

Development Plan

PR

QA

Please refer to all tests added, unit tests and also property tests (see test/unit/Cardano/Wallet/CoinSelection/Policy)

Context

Jörmungandr introduces a new binary block format. This format is described in
rust-cardano-/chain-impl-mockchain/doc/format.md.

In order to integrate with Jörmungandr, we therefore need to build a few
primitives first, starting with the block format. As a starter for testing,
we have the following test vector for the a genesis block generated via jcli:

005200000000009f000000000000000000000000ffadebfecd59d9eaa12e903a
d58100f7c1e35899739c3d05d022835c069d2b4f000000000000000000000000
00000000000000000000000000000000000000000047000048000000005cc1c2
4900810200c200010108000000000000087001410f01840000000a01e030a694
b80dbba2d1b8a4b55652b03d96315c8414b054fa737445ac2d2a865c76002604
0001000000ff0005000006000000000000000000000000000000000000000000
0000000000002c020001833324c37869c122689a35917df53a4f2294a3a52f68
5e05f5f8e53b87e7ea452f000000000000000e

NOTE:

Convert back to raw bytes using Data.ByteArray.Encoding(convertFromBase, Base16)

Decision

• Implement block decoder, in a modular fashion, similar to the existing
  implementation for the http-bridge (so this includes a transaction decoder,
  certificate decoder etc...)

• Disregard for now, anything related to transactions and only focus on the block structure and what's available in the genesis block.

Acceptance Criteria

1. We must have a type-1 (common structure) binary decoder for the new block format used by Jörmungandr
2. We must have a binary decoder that also supports decoding block headers in the format used by Jörmungandr
3. Type 2, Type 3, Type 4 and Type 5 binary decoders may be left for later.

Development Plan

Setup

• Create a new package
• Write a unit test, with stubbed undefined functions
• I plan on using the binary package
• Decode the genesis blockHeader and check/debug/log that values seem sane (e.g the decoded sizes-values are correct)

Then

• Decode Initial message
• Decode Transaction message
• ?

PR

• #244
• #245
• #266
• #273
• #275

QA

We have decoders for block headers, and the Initial [ConfigParam] and Transaction Tx (slightly incomplete) messages.

We test the decoders on one genesis block.

I think the plan for the future is to have more blocks to test against (after #242), and make the decoders more complete.

• The decoders are fairly strict.

  • There is a lot of size information, so we can use isolate extensively.
  • Throws on un-recognized constants/tags

• There is some intricate logic that we don't want to get wrong, but they are/should be covered easily by more blocks as test vectors.

Context

Golden tests can help us verify that our API json format doesn't change accidentally.

There were some talk about golden-tests on the legacy-wallet (and there was one ticket created input-output-hk/cardano-wallet-legacy#124)

API breaking changes are bad and also tricky to identify somehow. Right now, there's nothing really helping us not breaking the API apart from our own discipline. Despite being disciplined, something can also slip through our attention and get merged, unnoticed. A simple step towards this would be to add a few golden tests to make sure that API types aren't actually changing in a non backward-compatible ways.

If a given API type has one or a few associated golden test, hopefully, upon making a change, the underlying change would fail. Then, we can assess whether this change is a breaking change or just a soft extension.

We have / will have API-level types with Aeson ToJSON/FromJSON - instances. The API types can be composed of smaller types as product (record) or sum types. It seems like we could generate golden tests automatically.

Decision

Find a way to generate golden-tests automatically, to ensure that the JSON-format stays consistent. We want to test all interesting cases of our types (e.g make sure we test the format of all the error-cases in an error sum-type)

Acceptance Criteria

1. There must exist a way to trigger re-generation of golden-tests
2. Generated golden-tests should test the format of all API types, in all interesting cases. (May be weakened if infeasible)
3. We may utilise https://hackage.haskell.org/package/hspec-golden-aeson

Development Plan

• Play around with it, make pr
• Check how it handles large types (whether or not it adapts the number of samples to cover all cases) for second acceptance criteria

PR

QA

• See https://github.com/input-output-hk/cardano-wallet/blob/3c56a9ced1ffa122be37aacad96e983f1dc9b17b/test/unit/Cardano/Wallet/Api/TypesSpec.hs#L149-L161 for how it functions
• We need to be careful about missing cases of complex types. (cf #102 (comment)) Making sure to add tests for "sub-types" is also strategy to deal with this.

Context

In cardano-wallet-legacy we had integration tests written in a custom DSL. We want integration tests in this repo.

Decision

Port the DSL in preparation of adding integration tests here. The DSL is a long file, but we should only need the "core" parts of it.

Acceptance Criteria

1. We must identify appropriate parts of DSL.hs(and potentially related modules) for our new code-base
2. We must port these

Development Plan

• I intend(ed) to get the DSL module compiling
• New plan - #83 already seems like what we need here, so I think it would be enough to add 3cbf665 as a follow up.

PR

QA

• Verify that DSL.hs contains plausible functions for executing test scenarios, given that we don't have an API server in place, nor any code to start a dev cluster (for node backend to connect with).

Context

In the next Cardano era, address format is going to be different from what it is currently.
A specification of this revision is available on input-output-hk/implementation-decisions.

Note that in Shelley, addresses can now hold a staking key when they're involved into
delegation.

Another singularity of this new approach is the overall text-encoding of those addresses
which change from base58 to bech32.

Decision

• Implement a new version of the keyToAddress interface to support the new address format
  for both single and grouped addresses.

• Make sure that the address text-encoding is tight to the address format we chose (either
  review an existing type-class or create a new one to abstract the text-encoding from the
  wallet core)

Acceptance Criteria

1. We must support creating single address from a given public key using the new format

Development Plan

• Make sure address encoding is abstracted over the target backend
• Implement keyToAddress for Jormungandr backend
• Try to generate golden tests using jcli and have them as unit test in cardano-wallet-jormungandr ?

PR

QA

• PR #336 implements the EncodeAddress and DecodeAddress interfaces for Jormungandr. Jormungandr is slightly more complex than the cardano-http-bridge since it supports two different formats: bech32 (for new standard addresses) and base58 (for legacy addresses).

• In the test-suite, we have two kinds of tests. The first kind does standard round-trip tests to verify that decoding an address that we encoded works as expected. We have made sure to generate both kinds of addresses, being however slightly more biased towards the new format since the old one is already rather well tested in the http-bridge package. See:

  • The Arbitrary generator

  • The roundtrip test

• Added also a variety of negative tests to test various decoding paths.

  • Some are network-agnostic like here

  • Some actually depends on the target network. So we have them for:

    • Mainnet
    • Testnet
    • NOTE: there's no "staging" for Jormungandr

• Also generated a few addresses using jcli from the rust team, comparing results for both single and grouped addresses and also on both networks:
  - Mainnet
  - Testnet

• This gives us a 100% coverage on the EncodeAddress and DecodeAddress instance, with a pretty good confidence that we are able to encode and decode Shelley addresses correctly, might they be new or legacy ones.

Context

In order to test our wallet backend in a "real-life" scenario, we would need to be able to run a few integration tests where a the wallet server connects to a given chain producer backend in order to retrieve blocks.

For now, our only chain producer is the cardano-http-bridge and, it only connects to mainnet or testnet which is fine for testing a few things, but pretty limited when it comes to submitting transaction and moving test funds from some faucet wallet.

Having the ability to run a demo test cluster in CI and perform a serie of tests against it can be pretty useful.

Decision

First, we want to explore the capabilities of the cardano http bridge and figure out whether we can connect it to a local cluster. If this can be done with reasonable efforts, then we do it.

Acceptance Criteria

1. We must assess how complex it would be to instrument the cardano-http-bridge to run against a local cluster of node
2. If simple-enough, we may provide instructions and scripts to setup a local cluster to which the cardano-http-bridge could connect.

Development Plan

• Look whether it is possible to specify a custom configuration to the http bridge.

  Turns out it is with some minor changes, cf: input-output-hk/cardano-http-bridge#19

• Experiment connecting to the bridge to a local cluster

  I have tried a few things with the bridge and the local cluster and I am running into some issues. So far, here are my findings:

  • The bridge do use genesis data coming from the 'previous hash' as specified in the configuration. In order to do that, it seems that the bridge looks up some hard-coded initial data in exe-common/genesis. So far, only the data for the testnet, mainnet and staging are available.
  • Therefore, I've tried to spin up a local cluster using the same seed and genesis data. Yet, it seems that the protocol my local cluster uses is slightly different from what the bridge expects (the bridge expects the node to advertise its node id where the node seems to be sending raw data). I have quite figured out the problem here...

• Have a setup where it is possible to easily spawn a cluster and an http bridge connected to it.

PR

QA

• Integration tests can now able to start an integration cluster of cardano-node (using the latest release from cardano-sl) on top of which, we use the launcher code to start the wallet backend and the bridge

• I've updated the README with test instructions and pre-requisites for running the integration tests. Indeed, the cardano-node-simple and the cardano-http-bridge needs to be available on the global path.

• I've updated the dot travis file to make sure they're both available in CI:

  • https://travis-ci.org/input-output-hk/cardano-wallet/jobs/513019924#L259
  • https://travis-ci.org/input-output-hk/cardano-wallet/jobs/513019924#L266

• We do nothing with cluster at the moment, but I've added the plumbing code in the integration test to make sure the cluster is started (and killed) around the integration tests scenarios (cf: https://github.com/input-output-hk/cardano-wallet/blob/master/test/integration/Main.hs#L46-L73)

Context

We have:

• a ticking function (#3)
• a function to get blocks (#12)
• a running cardano-http-bridge subprocess (#8 -- in progress)

The ticking function should call the function to get blocks.

Decision

Add a function to get blocks every tick and log them.

startBlockSyncer
    :: MonadChainProducer m
    => NetworkName
    -> Port
    -> m ()

This function will be called from main and won't normally return.

Acceptance Criteria

1. Must be able to catch up with blocks from chain producer.
2. Must process (by logging to stdout) the blocks received.

Development Plan

• Implement startBlockSyncer
• Call startBlockSyncer from main`
• Add iohk-monitoring-framework and use the basic set up which they have documented. (Maybe not yet -- haven't decided)
• Log new blocks at "info" level
• Rely on an externally launched cardano-http-bridge
• Reflect upon design of the receive and process blocks feature, and propose then implement changes.

PR

QA

• Okay, we do now connect and receives blocks from the http bridge, pretty much epoch per epoch. Note that this is very very rudimentary for now and we are missing quite a few features coming soon:

  • Ability to persist the progression (ideally, we do want to process blocks as we fetch them, and store the latest tip we've processed).
  • Handling rollbacks and duplicates

• Still, this can be manually tested by starting the current "wallet server" alongside the http bridge. Provided you've installed in cardano-http-bridge (cargo install https://github.com/input-output-hk/cardano-http-bridge.git):

cardano-http-bridge start --port 1337 & 
stack exec -- cardano-wallet-server --http-bridge-port 1337

This should print all blocks received from the http bridge, until the tip block is reached, whereupon a network block should be printed every 20 seconds.

• Also, check what happens if the node backend crashes (or gets killed).

Context

As per the specification, there are various types and low-level primitives we may define for the wallet (about UTxO manipulation, block application etc).

Decision

• A module must define primitive types to represent a wallet in its very core
• A module must define primitive operations on those types
• We must also implement the various properties and proof available in the wallet specifications

Development Plan

PR

QA

• This was a bit longer than initially planed. In the end, two modules were added to the source code:

  • Cardano.Wallet.Primitive
  • Cardano.Wallet

• Both modules have corresponding property tests

  • For Cardano.Wallet.Primitive, I have translated to Haskell the various Lemma and properties available on those primitive types in the specification for a Cardano wallet (cf: https://github.com/input-output-hk/cardano-wallet/blob/eb3a993e6b58ce459e4320c6a34ba0d5b2399bf4/test/unit/Cardano/Wallet/PrimitiveSpec.hs#L58-L86).

  • For Cardano.Wallet, I've implemented in test the Basic Model from the specification, and compared the tracking of UTxO between that basic model and our slightly more complex model which includes prefiltering -- making sure they match. As we add features to the wallet code, this may become quite useful to make sure that we aren't breaking the basic capabilities of keeping track of UTxO and a balance. I've also added there another Lemma from specification which makes sure that our basic model implementation from the test code actually matches what the specification expect. cf: https://github.com/input-output-hk/cardano-wallet/blob/master/test/unit/Cardano/WalletSpec.hs#L79-L84

Context

Bug to gather and address some issues that need to be fixed for CLI

Need to be fixed:

• get wallet doesn't work, prints out Something went wrong fixed by #230
• --version doesn't print out anything
• --port is missing on delete wallet
• delete wallet does not print anything when you put id that does not exists and prints out just [] after successful deletion. I'd expect some more informative messages (e.g. Wallet not found and Wallet deleted successfully respectively)
• cardano-wallet get wallet --wallet-id=457570f4c6d7fceff6f6945711e710f3ea008433 - wallet id that does not exist prints out nothing (same as for delete wallet), again something like Wallet not found would be better

Might be addressed here
(These are not really bugs but some ideas and at most usability flaws. Some might be addressed here. Anything not addressed here should land on the list of Ideas - however let's discuss before tackling anything from here, as there are ongoing discussions on whether we still stick to docopts)

• I agree with @rvl to make wallet-id a positional argument (especially as it is now not consistent --wallet-id (on get wallet) vs. --id (on delete wallet))
• I like the @jonathanknowles idea, that would organize things nicely (I had similar approach on cardano-wallet-legacy when created a script for aiding some manual scenarios https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/test/manual/scripts/wallet.sh#L3-L11, btw. wallet-id is also a positional arg there, and in particular for wallet get could be optional wallet get [<wallet-id>])
  One alternative that I can think of is put the noun first:

cardano-wallet wallet get
cardano-wallet wallet list
cardano-wallet wallet create
cardano-wallet wallet delete
cardano-wallet mnemonic generate

This would have the advantage that we could support different "levels" of help. For example:

cardano-wallet --help
cardano-wallet wallet --help
cardano-wallet wallet list --help

• I like the idea that passphrase is hidden with * but I think there needs to be another prompt for repeating the password (as pointed by @KtorZ)
• for mnemonics I think I'd rather leave them not hidden with *. I think if someone would try to write them manually it'd be very easy to make a mistake (as already pointed). In practice, when creating new wallet, I think it would be rather a copy-paste thing from places like https://iancoleman.io/bip39/ or simply copying output from cardano-wallet generate mnemonic therefore, again, hidding them does not seem to be very useful.
• generally I think that create wallet could be made a responsive CLI for all arguments for consistncy (currently --name and --address_pool_gap remain as inline arguments)

Steps to Reproduce

stack exec -- cardano-wallet

Expected behavior

Actual behavior

Resolution Plan

• (@KtorZ - #230) I'll start investigating the issue with the get wallet command
• (@KtorZ - #232) Rework the CLI hierarchy to put resources first and add missing options
• (@KtorZ - #232) Implement --version using the version mentioned in the .cabal file
• (@KtorZ - #232) Make wallet id argument positional
• (@KtorZ - #232) Implement better response handling for 404 responses and delete success
• (@KtorZ - #232) Prompt passphrase twice for repetition
• (@KtorZ - #232) Do not hide mnemonic when typed
• (@KtorZ - #232) Do now show ANSI colors if not a terminal

PR

• #230
• #232

QA

I think every points in "should be fixed" and mostly everything in "might be fixed" have been addressed. See #232 for the complete list of things that were done.

Context

At this moment we follow pending transaction set. We need to account also for confirmed transactions. In order to do it we need to extend the wallet state by the list of transaction. Caution should be put here as this extension (in contrast to pending transaction set) is not covered in the wallet spec. There may be the need to extend the wallet spec because of that. Moreover, the keeping tract of known transactions should not affect already established invariances and laws following from wallet spec.

Decision

We will add known transactions to the wallet state.

Acceptance Criteria

1. The known transaction list must be added to wallet state
2. The extension must not break anything from the wallet spec
3. Known transaction should be investigated for possible nontrivial laws and invariances with possible wallet spec extension.

Development Plan

• Look in to how wallet state is updated with applyBlock
• I have decided to add a field to Wallet rather than to the Wallet state parameter.
• Figure out some properties

PR

QA

• We have extended our primitive model to now keep track of discovered transactions during block applications. These transactions are then returned as outputs rather than being stored in the wallet state (as they can grow out-of-bounds) and stored separately from the wallet state itself.

• We have manually compared the results of restoring an existing known wallet from mainnet in Yoroi and in our current backend (I can share the mnemonic if interested to compare and tests). This wallet is rather useful as it contains both incoming and outgoing transactions, as well as transactions with no change. So it allows us to test various edge-cases of the discovery process.

• Alongside that, we've also extended our automated test-suite and, in particular, our test data in #148 such that we would generate better "fake" blockchain with both incoming and outgoing transactions.

Context

The way the master key got generated from a mnemonic sentence has changed between the legacy random address scheme, and the new sequential scheme on Cardano. We do now have the possibility to add an extra recovery passphrase during the seed / master key generation which adds an extra protection on the mnemonic (having the mnemonics isn't enough, one would also need to know the recovery passphrase).

This is supported by cardano-crypto through the generateNew function as shown here:

https://github.com/input-output-hk/cardano-crypto/blob/3c5db489c71a4d70ee43f5f9b979fcde3c797f2a/src/Cardano/Crypto/Wallet.hs#L124-L129

Decision

We want to extend our Cardano.Mnemonic module to support having a recovery passphrase that may be set by a user via the API.

Questions

1. What relevance has this to the Cardano.Mnemonic module? I thought that the recovery passphrase would be only used when generating the root private key.
2. Will a "spending password" still be used to encrypt the root private key?
3. Will the spending password and recovery passphrase be the same?

Development Plan

PR

QA

Context

As part of this rewrite of the wallet backend, we do not want to depend on any of the code from cardano-sl; Yet, there are a few bits that were made in a fairly isolated manner that are still very relevant to the wallet This is the case of:

https://github.com/input-output-hk/cardano-sl/tree/develop/mnemonic

Minus a few bits that are strictly related to cardano-sl like:

• https://github.com/input-output-hk/cardano-sl/blob/develop/mnemonic/src/Cardano/Mnemonic.hs#L155-L186
• https://github.com/input-output-hk/cardano-sl/blob/develop/mnemonic/src/Cardano/Mnemonic.hs#L237-L240
• https://github.com/input-output-hk/cardano-sl/blob/develop/mnemonic/src/Cardano/Mnemonic.hs#L286-L316

Decision

We do want to port this module onto cardano-wallet, review the style to make it consistent with our coding standards, and remove the bits that aren't relevant to this new code. We do want to make this a module of cardano-wallet (and not a package as it is now), under Cardano.Wallet.Mnemonic.

Tests should also share the same faith and be ported in their rightful place.

Development Plan

PR

QA

The PR introduces Cardano.Wallet.Mnemonics module which delivers mnemonic creation. From the point of view of testing there are two aspects :
(a) unit tests that are in located in Cardano.Wallet,MnemonicsSpec module.
(b) executable that uses this and generates Mnemonic 15 mnemonics

$ stack build
[pawel@arch cardano-wallet]$ stack exec cardano-generate-mnemonic
["love","dutch","usual","uphold","whale","certain","merry","horror","bulb","injury","school","crane","grass","police","beauty"]

Context

During the development of v0.0.1, we have accumulated some technical debts we may get rid of while getting started with this next release.

Poor README

Areas missing testing:

Unused code

Integration framework

Decision

From the various points stated above, evaluate whether they are relevant and, fix them when they are, or leave an appropriate comment in the code giving details or insight about the decision.

Acceptance Criteria

1. We must have looked into the various points above, and either tackled them or discarded them with a proper justification.

QA

Context

After we figured out #67, #55 and #56 we should have all the keys in hand in order to run integration tests for our wallet, testing the end-2-end connectivity between a cluster of node, our wallet backend and an http client making request to the wallet server.

Decision

Implement simple example of integration tests which illustrate a complete setup running tests against a local cluster. We may get inspiration from cardano-wallet-legacy

Then, make sure those tests runs in CI as expected, and provide a correct report coverage. For the coverage report, we may look into hpc combine in order to combine tix files from the unit tests and the integration tests (allowing both jobs to run in parallel rather in sequence).

Acceptance Criteria

• We must be able to run integration tests through stack test cardano-wallet:integration
• Integration tests must run in the CI and, block code from being merged when failed
• Integration tests may be run in parallel (through a separate job) with unit tests
• Integration tests results should also be part of the coverage reports

Development Plan

• Define a basic structure for integration tests
• Add some integration tests connecting to the cardano-http-bridge on testnet
• Add necessary steps in Travis and find a way to cache (part of) the testnet blockchain data

PR

QA

• We now have a folder and cabal target for integration tests (stack test cardano-wallet:integration). A few gotchas however:

  • Integration tests require the cardano-http-bridge to be available in scope
  • The HTTP Bridge should have been (partially -- at least the first 15 epochs) synced with testnet

• The CI has been adjusted to now download an extra cache containing the blockchain data for testnet (for the first 15 epochs) and integration tests are now ran as part of the standard test-suite (which means that they also count for the coverage report). We may split them into two separate jobs later if you judge it necessary.

• I've also reviewed a bit our deployment stage on Travis for it wasn't really working well. I've made a few test on a dummy repository of my own to make sure that we get the following behavior to work:

  • Upon releasing, one can create a git tag on git tag --sign -m "release v.X.Y.Z" vX.Y.Z master
  • Pushing a tag to CI will now trigger a deployment git push origin --tags, which will run all automated tests and, deploy the launcher and wallet server executables to our github releases
  • Github releases are made in a "draft" mode, such that they aren't visible by anyone outside of the team and can be edited (adding a description or CHANGELOG).
  • When ready, we can simply press the "publish release" button in Github to make the release available to the public.

Context

As part of setting up repo for wallet rewrite.

Decision

[x] Every part of cardano-wallet team must get merge permission.
[x] Merge permission must be available if:

• at least one reviewer accpeted the PR
• our CI is green

NOTE: Reviews will be done mostly in pairs (pair-programming) with dev rotation around the code so that everyone is familiar with bits of the whole system.

Development Plan

Explained in the section above

PR

None.

QA

Done. See: settings/branches

(Not sure if one needs admin access to see these settings but, just in case, for what it's worth 😂 :)

$ git push origin master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 711.10 KiB | 0 bytes/s, done.
Total 5 (delta 0), reused 0 (delta 0)
remote: error: GH006: Protected branch update failed for refs/heads/master.
remote: error: At least 1 approving review is required by reviewers with write access.
To [email protected]:input-output-hk/cardano-wallet.git
 ! [remote rejected] HEAD -> master (protected branch hook declined)
error: failed to push some refs to '[email protected]:input-output-hk/cardano-wallet.git'

Context

We intend to maintain both a Swagger API specification and a Servant API specification, and to keep the two in synchronization with one another.

Task

Translate the Swagger API specification (specifications/api/swagger.yaml) into an equivalent Servant API specification, including:

• Types
• Endpoints

Acceptance Criteria

1. Our Servant API must satisfy the Swagger API specification. Clients generated by the official Swagger Codegen tool should be able to connect and make API calls without issues.
2. Endpoints defined in the Servant API may include specifications of errors that may be thrown.

Development Plan

• Code up a small portion of the API, with at least one endpoint, and all the types required to support that endpoint.
• Extend the API to support all endpoints listed below: (see QA section)
• Translate the postWallet and putWallet operations to the Servant API, along with all associated types.

PR

QA

• The scope of this ticket has been reduced down to the endpoints discussed during the last iteration planning meeting:

  • List wallet
  • Get wallet
  • Post wallet
    (- Delete wallet) done as bonus here

• Transaction creation and listing will be covered by #93

• Translated types are available in the haddock documentation here: https://input-output-hk.github.io/cardano-wallet/haddock/cardano-wallet-2.0.0/Cardano-Wallet-Api-Types.html.
  The strategy here was to re-use most of our low-level types and wrap them in a polymorphic ApiT to define the various serialization instances we needed. This way, we keep the API layers fairly separated from the rest and we avoid cluttering the core code with API-related concerns.

• For now, there's no server actually implementing these types, but we have been already able to perform quite some extensive testing (cf: #91). Testing can be found in Cardano.Wallet.Api.TypesSpec and are of four kinds:

  • Roundtrip tests for JSON for types appearing in the API

  • Golden tests for JSON representation of types appearing the API (prevent breaking change), cf #91

  • Comparison of every response & body param type of the API against their corresponding swagger representation (so here, Wallet, [Wallet] and WalletPostData). It works by generating arbitrary samples for each types, and then, checks whether they satisfy their corresponding schema from the spec. So, it was possible to represent a value in Haskell that would violate one of the constraints we have in swagger, the test would fail.

  • And finally, tests which check that all paths present in our API definition also exist and match one from the specification. We do not do the other direction (verifying that every path in the spec exist in our API) because we obviously haven't implemented everything yet, though in practice, it would be a nice one to have later.

• Added a few negative test cases to test failing paths in various decoders that aren't fully generic (see: #128)

Context

We want to have CI set up for the new cardano-wallet in order to have a means to automatically build the code and run all our tests against the build. At this moment we decided to go with Travis, so we want to get the .travis.yaml from the previous cardano-wallet review it and remove all unnecessary dependencies to cardano-sl from it. This task is also to make sure that the new repo is connected to Travis CI.

Decision

Connect new cardano-wallet repo with Travis CI. Get the .travis.yaml from the previous cardano-wallet review it and remove all unnecessary dependencies to cardano-sl from it. This task is also to make sure that the new repo is connected to Travis CI.

AC

• Travis is running builds on PRs.
• Travis is running builds on the master branch.
• Tests are being executed by Travis.

Development Plan

• Initial Travis build which runs stack test.
• Connect repo to travis.
• Also run Coveralls and stylish-haskell.

PR

QA

• Check that the list of AC is complete and done.

Context

While doing POST v2/wallets one needs to provide address_pool_gap which needs to be between 10 and 100.
When providing the value that is outside range there is an error message thrown Error in $['address_pool_gap']: ErrGapOutOfRange. This message should be made more informative and human friendly.

Steps to Reproduce

1. start cardano wallet launcher:

export NETWORK=testnet
stack exec -- cardano-wallet-launcher

curl  -vX POST http://localhost:8090/v2/wallets \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["identify", "screen", "lock", "bargain", "inch", "drop", "canyon", "flock", "dry", "zone", "wash", "argue", "system", "glory", "light"],
"passphrase": "Secure Passphrase",
"address_pool_gap": address_pool_gap
}' --http1.1

case 1
address_pool_gap = -1

case 2
address_pool_gap = 20.5

case 3
address_pool_gap = 101

Expected behavior

case 1, case 2, case 3

Error in $['address_pool_gap']: Provided value is outside of range [10..100]

Actual behavior

case 1

Error in $['address_pool_gap']: Word8 is either floating or will cause over or underflow: -1.0

case 2

Error in $['address_pool_gap']: Word8 is either floating or will cause over or underflow: 20.5

case 3

Error in $['address_pool_gap']: ErrGapOutOfRange 101

Resolution Plan

• It should be enough to modify the 'FromJSON' instance from the Api types to make it use the 'FromText' instance which already has a nice error message.

PR

QA

• Tests updated in Cardano.Wallet.Api.TypesSpec to illustrate the new message.

• Also, did some manual verifications with cURL

Scenario: bigger than the maximum

 $ curl  -vX POST http://localhost:8090/v2/wallets   -H "Content-Type: application/json; charset=utf-8"   -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["identify", "screen", "lock", "bargain", "inch", "drop", "canyon", "flock", "dry", "zone", "wash", "argue", "system", "glory", "light"],
"passphrase": "Secure Passphrase",
"address_pool_gap": 250             
}' --http1.1

< HTTP/1.1 400 Bad Request
Error in $['address_pool_gap']: An address pool gap must be a natural number between 10 and 100.

Scenario: smaller than the minimum

$ curl  -vX POST http://localhost:8090/v2/wallets   -H "Content-Type: application/json; charset=utf-8"   -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["identify", "screen", "lock", "bargain", "inch", "drop", "canyon", "flock", "dry", "zone", "wash", "argue", "system", "glory", "light"],
"passphrase": "Secure Passphrase",
"address_pool_gap": -5 
}' --http1.1
< HTTP/1.1 400 Bad Request
Error in $['address_pool_gap']: An address pool gap must be a natural number between 10 and 100.

Scenario: not a number

$ curl  -vX POST http://localhost:8090/v2/wallets   -H "Content-Type: application/json; charset=utf-8"   -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["identify", "screen", "lock", "bargain", "inch", "drop", "canyon", "flock", "dry", "zone", "wash", "argue", "system", "glory", "light"],
"passphrase": "Secure Passphrase",
"address_pool_gap": "patate"
}' --http1.1

< HTTP/1.1 400 Bad Request
Error in $['address_pool_gap']: expected Int, encountered String

Context

As a wallet user I want an option when sending a transaction to specify the change address. example: I have utxo with 100 Ada. I send 10 Ada to someone. Currently a new address is generated and 90 ADA goes into that address. But I want to be able to specify my-self the address that received the 90 ADA change.

Decision

Acceptance Criteria

Development Plan

PR

QA

Context

We have:

• A set of core primitives and types to represent and keep track of a wallet
• A network layer that retrieves blocks from a chain producer (i.e. the cardano-http-bridge)
• A db layer that define an interface for persisting pieces of information to some db engine / storage
• A wallet layer that connects the dots and make use of the above to provide a simple interface for wallet clients (like the API layer or the CLI)

Decision

We do need to implement the wallet layer to make sure that we process blocks from the chain producer and apply them as we received them. Also, this should ideally make use of the DB layer to keep track of the checkpoints and, restart from a given position when needed.

Acceptance Criteria

• The wallet layer must process blocks from the NetworkLayer using the wallet primitives
• The wallet layer should flush its latest state to the database after processing blocks
• The wallet layer may quickly catch up with the network tip when started

Development Plan

• Draft a first pipeline that tight together the various layers in order to restore a wallet from a given mnemonic.
• Look at the heap and time profiling for restoring a wallet against mainnet
• Write test code for the DBLayer
• Write test code for the WalletLayer

PR

QA

• We've finalized the first draft implementation of the wallet layer as described by WalletLayer. To implement this layer, we had to pull strings from the DBLayer and NetworkLayer, ultimately tightening them together alongside the wallet primitive logic to accomplish basic wallet work.

• Usage of the watchWallet have been manually tested and profiled and the result can be seen in comment of #73 (note that, I have later removed the corresponding code from the CLI executable as it seemed to confuse people).

• @paweljakubas also added extra unit tests to test basic properties on our fake DB engine for now, that are visible in Cardano/DBLayer/MVarSpec.hs

• We do also have similar tests working directly at the WalletLayer level and checking additional invariant (like, conflict between ids), cf: Cardano/WalletLayerSpec.hs

Context

The wallet requires an underlying node. We should have a single command for starting both correctly.

Decision

• Support the following CLI parameters
  • --wallet-server-port (the port we are serving the wallet API on)
  • --node-port (the port were communicating with the node on)
  • --network (mainnet or testnet)

Development Plan

PR

QA

Tested manually. See PR description of #27 for examples.

Context

Sub-issue of issue #53 (Translate the Swagger API specification into a Servant API specification).

Decision

Translate the postWallet and putWallet operations to the Servant API, along with all associated types.

Acceptance Criteria

1. The Servant API must support the postWallet and putWallet operations.

PR

PR #119

Context

Adding support for transactions. In a separate issue (to be created), Transaction-related types will be introduced inside WalletLayer. This ticket is about modeling this functionality on the API-level — creating wrapping types, JSON instances, etc.

• Cf https://github.com/input-output-hk/cardano-wallet/wiki/Roadmap#submit-transactions for some additional details

• The API we want to implement is described by our swagger spec https://raw.githubusercontent.com/input-output-hk/cardano-wallet/master/specifications/api/swagger.yaml

• Note: The API layer is under construction in #53 as of currently

Decision

Add wrapping Transaction-related API types, and extend our Servant API with handlers.

Acceptance Criteria

1. The API must support creation of transactions according to the API Swagger specification https://raw.githubusercontent.com/input-output-hk/cardano-wallet/master/specifications/api/swagger.yaml
2. The added code should be consistent with existing API Layer code (likely from #53 )

Development Plan

• I plan on adding new transaction endpoint to the Cardano.Wallet.Api module
• I plan on adding missing argument types (Payment, TransactionOutput) to Wallet.Api.Types module
• I plan on adding missing return type (Transaction). This type looks a lot a like transaction meta which will be added by #101
• I plan on adding json roundtrip test for arguments and return type
• I plan to remove swagger object renaming rules #111 (comment)
• I plan to increase number of samples for tx status golden test #111 (comment)
• I plan to document types better #111 (comment)
• TODO: check whether there are other tests that can increase coverage

PR

QA

• API definition is now complete for transaction creation: cf Cardano.Wallet.Api

• Corresponding API representations have corresponding golden tests roundtrips in Cardano.Wallet.Api.TypesSpec

• We do also check that the servant definition matches the API specification (cf: here)

• The handler's implementation is currently untested but will be as part of #96 or #95

Context

In a similar fashion to how the cardano-http-bridge works, Jörmungandr offers a
REST API to work with. This API is still in progress / under development but
already offers a few endpoints we can start integrating with in order to communicate
with the node.

A swagger specification for this API is available here (rendered)

Decision

• Implement a new instance of our NetworkLayer for Jörmungandr, using the API provided in Context

Acceptance Criteria

1. The getBlocks from our NetworkLayer must be implemented using Jörmungandr REST API
2. The networkTip from our NetworkLayer must be implemented using Jörmungandr REST API
3. The postTx from out NetworkLayer must be implemented using Jörmungandr REST API

Development Plan

General

(@Anviking vvv)

• Define the Servant Api type
• The Network module needs to be implemented. Some parts might be non-trivial as it differs from http-bridge.
  • Implement JormungandrLayer (excluding postTx)
  • Implement NetworkLayer
    • Change the wallet tip to (SlotId, BlockId)
    • Should then be able to implement a Jormungandr backed NetworkLayer
    • It would be… good… to have more blocks to test getDescendants. I haven't managed to get some yet. #382

(@KtorZ vvv)

• Extend BlockHeader definition to include the current block hash
• Compute corresponding block header hash for Jormungandr implementation
• Review how initWallet received its first blockHeader ....

signTx

Signing might be some substantial amount of work

• See if we can submit transactions to jormungandr using the old signing schema, in a minimum viable way. We'd have to add encoders for transactions and witnesses

Finalizing

• Review code and add more tests

PR

QA

• We have implemented a NetworkLayer for Jörmungandr using an underlying JörmungandrLayer. We added some integration test cases covering networkTip, nextBlocks and postTx. They are not able to test that transactions get accepted by the node. I believe that is tested by #358 though. https://github.com/input-output-hk/cardano-wallet/blob/master/lib/jormungandr/test/integration/Cardano/Wallet/Jormungandr/NetworkSpec.hs#L91-L225
• We fixed flaws with the Jörmungandr.Binary module (verified by working integration tests) and goldens
• We made encoders "safer" by throwing instead of overflowing when converting integers #487. The coin-selection shouldn't create that large txs, so we should never hit those cases, but it's good to be safe.
• We added a TxId class and comparing txId with jcli-generated golden tests: https://github.com/input-output-hk/cardano-wallet/blob/a2f24584b3a1015487ecadff523bd89242024616/lib/jormungandr/test/unit/Cardano/Wallet/Jormungandr/CompatibilitySpec.hs#L112-L157
• We implemented the Jörmungandr TransactionLayer and have golden tests comparing jcli with mkStdTx: https://github.com/input-output-hk/cardano-wallet/blob/a2f24584b3a1015487ecadff523bd89242024616/lib/jormungandr/test/unit/Cardano/Wallet/Jormungandr/TransactionSpec.hs#L225-L360
• We have added Binary roundtrips for signed transactions and addresses #476
• We were forced to change the type of the wallet and network tip from SlotId to BlockHeader.BlockHeader contains slotId and a prevBlockHash. Jörmungandr needs prevBlockHash and HttpBridge needs slotId. Should be covered by existing Network-tests and DB-related tests.
• Aside from one initially, I believe the integration tests (#358 ) hasn't discovered any problems with this code, giving us some more assurance.

Context

The cardano-http-bridge has a rather basic API where data can be retrieved in a binary format. The format used is a mix between the format used on a cardano-sl (a custom-made encoding / decoding protocol built on top of CBOR and some custom packfile created by the Rust team.

See https://github.com/KtorZ/wallet-prototype/blob/master/app/Main.hs#L973 for starter.

Decision

We want the ability to at least decode those binary data in order to be able to use the cardano-http-bridge as an initial middle-man.

• We must be able to decode our wallet primitive types from CBOR
• We should use the cborg library from hackage (same one used by cardano-sl) to minimize the risk with implementation details
• We may discard unnecessary information from blocks and only retrieve what we need

Development Plan

• Look at cardano-chain repo for inspiration.
• Copy&paste decoders implemented in the prototype.
• Add golden tests for decoders.
• No need to add ToCBOR/FromCBOR typeclasses yet.
• Implement rust-cardano packfile routines.
• Implement encoders in another issue (e.g. #42) and add roundtrip tests where there are also encoders defined.

PR

QA

• These are covered by unit tests.
  • Cardano.Wallet.BinarySpec - These are golden tests based on blocks and block headers found on mainnet and testnet.
  • Cardano.Wallet.Binary.PackfileSpec - These are testing first some error cases with synthetic pack file data, then a golden test using epoch 104 of mainnet.
• Ensure tests pass and coverage is adequate.
• Tests cover empty blocks, blocks with a single transaction, blocks with multiple transactions and invalid blocks.

Context

We need the launcher that enables us to start the cluster comprised of the wallet and cardano-htttp-bridge that is intermediary between the wallet and mainnet/testnet.

Decision

We will enable launcher starting the cluster that consists of wallet and http bridge instance. As the http bridge is the gateway for block acquisition we will spawn two instances together as parts of lauhing, but also kill the launcher if any part of the cluster is down.

Development Plan

PR

QA

• We now have two executables available

  • cardano-wallet-server: which only spawns a wallet server that connects to an http-bridge
  • cardano-wallet-launcher: which spawns a wallet launcher that spawns and supervise both the cardano-http-bridge and the cardano-wallet-server.

• Note that both executables expect the cardano-http-bridge to be "in scope", so it should have been installed (cargo install --git https://github.com/input-output-hk/cardano-http-bridge.git) and, the $PATH env var should at least point to $HOME/.cargo/bin.

• There are also some unit tests to verify that the launcher works correctly, meaning, that it stops both processes if one of them exits. When stopped, the launcher also cleans up behind it and remove allocated resources (see: Cardano.LauncherSpec)

• We have also added an extra signal handler on UNIX system such that, the launcher will exit properly (and kill its child processes) when it receives a SIGTERM. The manual procedure for this is described in: test/manual/Cardano/Launcher/POSIXSpec.md

Context

We have added coin selection and fee calculation in the context of the former.

Decision

We need to add fee estimation. Fee estimation is determined based on bytes imprinted in transaction send to the node. The estimation can be relatively simple as:
(a) Inputs are always of the same size because they're a hash and index
(b) Outputs have variable sizes, but that can be easily determined regarding their value (<24 one byte, <256 2 bytes, <65536 3 bytes etc ... ). The additional variable is derivation scheme adopted.
(c) tx witnesses, one per input, are also fixed-sized
(d) there is fixed-sized cbor bits (connected with building lists)
(e) network can be mainnet or testnet

Acceptance Criteria

1. We must be able to adjust a coin selection for fee
2. We must estimate the size of a (Byron) transaction accurately
3. Estimated size must always be greater than or equal to the actual size
4. Fee estimation may not be exact, but ideally with a error of <1%

Development Plan

PR

QA

See extensive tests in Cardano.Wallet.CoinSelection.FeeSpec. We test mostly on two fronts:

• The adjustForFee function is tested via a few hand-crafted unit tests. The function is in practice non-deterministic (runs in MonadRandom m => ...), though, with a small number of available inputs or, by making them all the same, we can "predict" what input will be selected.

• On the other hand, we do also check a few properties for this function (see here):

  • No fee gives back the same selection, when fees are null
  • Fee adjustment is deterministic when there's no extra inputs (because indeed, we only need the 'MonadRandom' to randomly select a new input. If there's none, we can either adjust without any extra inputs, or we fail; but the algorithm is always deterministic).
  • Adjusting for fee (/= 0) reduces the change outputs or increase inputs; that's the basic intuition we can have of this function, it divvies the fee over all change outputs and decrease them by the requested amount. If there's not enough change to cover for fees, then a new input is selected.

• On the other hand, we do also compute fees for a given coin selection. This is rather "tricky" because fees depends on the underlying binary representation for a transaction (cf: comments here). So, we do have a property checking that "Estimated fee is the same as taken by encodeSignedTx", which computes, for any given coin selection, the actual true fee for the corresponding transaction. Our estimation has to be rather "identical" (which is not possible in practice, because we can't know upfront the size of the change address. In practice, sequential address payloads on mainnet will vary between 39 & 43 bytes, so we consider equality here with a margin of 4 bytes per change output. Note that we also run the property a thousand time because the generator work in a rather big input domain (up to 100 inputs and outputs; because many of the CBOR encodings change / increase when list length get bigger than 23!). Hence, we check that estimated fee are at least as big as if exact, and, we allow them to be bigger than the actual fee up-to 4 bytes per change output: https://github.com/input-output-hk/cardano-wallet/blob/master/test/unit/Cardano/Wallet/CoinSelection/FeeSpec.hs#L380-L388

Context

Similarly to #21, we do want to port existing logic from cardano-wallet-legacy onto cardano-wallet, from the following modules:

• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/src/Cardano/Wallet/Kernel/AddressPool.hs
• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/src/Cardano/Wallet/Kernel/AddressPoolGap.hs
• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/test/unit/Test/Spec/AddressPoolGap.hs
• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/test/unit/Test/Spec/AddressPool.hs

Also, we did initially embed in AddressPool some logic to create new addresses. This could actually be removed in order to make the AddressPool a plain data-structure that can be easily serialized / deserialized (much more like done in: https://github.com/KtorZ/wallet-prototype/blob/master/app/Main.hs#L719-L730

We do not need any particular abstraction on how addresses are derived at this stage and we can only consider AddressPool and AddressPoolGap in the context of sequential derivation with BIP-44.

Decision

• We must port modules described above to cardano-wallet and remove any dependency to cardano-sl
• We should merge AddressPool and AddressPoolGap into one module
• We should remove unnecessary abstraction over the address derivation (as described above)

Development Plan

• I intend to import and merge together the AddressPoolGap and AddressPool modules from cardano-wallet, alongside with their testing code.
• I plan to align types with primitives and wrappers provided in AddressDerivation & Cardano.Wallet.Primitive
• I intend to make AddressPool a plain data-structure (as it is in the wallet prototype) to be more idiomatic haskell.

PR

QA

• The module define two main data-structure (cf haddock

  • AddressPoolGap: to represent the number of consecutive undiscovered addresses
  • AddressPool: to keep track of a set of addresses for a given account and change chain

• Properties are define for both structures as follows:

  • AddressPoolGap
  • AddressPool
    The ones for AddressPoolGap are rather straightforward and just make sure that our wrapper type has the correct properties and define correct bounds. Not that, I've removed the Num, Real and Integral instances from the legacy code because this was just giving too much power to the wrapper, and wasn't really needed after all. An Enum instance is enough in most cases.

Context

Error message for mnemonic_sentence and mnemonic_second_factor need to be improved:

E.g. when doing POST v2\wallets and providing invalid mnemonic sentence or valid mnemonic sentence but with invalid length to one of the parameters the following error is returned:

Error in $['mnemonic_sentence']: ErrMnemonicWords (ErrWrongNumberOfWords 12 24)

• the error is not really human friendly and can be misleading
• it is faultly thrown for the case when the number of words is valid but mnemonics are invalid
• it is faulty thrown for the non-english mnemonics (ErrMnemonicWords (ErrWrongNumberOfWords instead of ErrDictionary (ErrInvalidDictionaryWord)

Steps to Reproduce

Please note that examples are for mnemonic_sentence but the same situation adheres to mnemonic_second_factor.

0. start cardano wallet launcher:

export NETWORK=testnet
stack exec -- cardano-wallet-launcher

Case 1 - invalid mnemonics, valid length

1. Run post wallet with invalid mnemonics list, yet make sure list has valid length (e.g. 15)

curl  -vX POST http://localhost:8090/v2/wallets \
  -H "Accept: application/json; charset=utf-8" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["word","word","word","word","word","word","word","word","word","word","word","word","word","word","word"],
"passphrase": "Secure Passphrase",
"address_pool_gap": 20
}' --http1.1

Case 2 - valid mnemonics, invalid length

1. Run post wallet with valid mnemonics list, yet make sure list has invalid length (e.g. 9)

curl  -vX POST http://localhost:8090/v2/wallets \
  -H "Accept: application/json; charset=utf-8" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["subway", "tourist", "abstract", "roast", "border", "curious",
        "exercise", "work", "narrow"],
"passphrase": "Secure Passphrase",
"address_pool_gap": 20
}' --http1.1

Case 3 - valid, yet non-english mnemonics

curl  -vX POST http://localhost:8090/v2/wallets \
  -H "Accept: application/json; charset=utf-8" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
"name": "Piotr Wallet",
"mnemonic_sentence": ["盗", "精", "序", "郎", "赋", "姿", "委", "善", "酵",
        "祥", "赛", "矩", "蜡", "注", "韦", "效", "义", "冻"],
"passphrase": "Secure Passphrase",
"address_pool_gap": 20
}' --http1.1

Expected behavior

Case 1 - invalid mnemonics, valid length
The error should be rather something like:
Error in $['mnemonic_sentence']: The list of provided mnemonic words is invalid.

Case 2 - valid mnemonics, invalid length
The error should be rather something like:
Error in $['mnemonic_sentence']: The list of provided mnemonic is too short (9). Supported lengths are 15,18,21,24.

Case 3 - valid, yet non-english mnemonics
Error in $['mnemonic_sentence']: Non-english mnemonic words are not supported

Actual behavior

Case 1 - invalid mnemonics, valid length

The error message returned for the case is:
Error in $['mnemonic_sentence']: ErrMnemonicWords (ErrWrongNumberOfWords 15 24)

Case 2 - valid mnemonics, invalid length
The error message returned for the case is:
Error in $['mnemonic_sentence']: ErrMnemonicWords (ErrWrongNumberOfWords 9 24)

Case 3 - valid, yet non-english mnemonics
Actually the same error is returned -> Error in $['mnemonic_sentence']: ErrMnemonicWords (ErrWrongNumberOfWords 9 24) instead of expected ErrDictionary (ErrInvalidDictionaryWord .... However even the ErrDictionary (ErrInvalidDictionaryWord ... should be polished to be informative and user-friendly as indicated in Expected behavior.

Resolution Plan

• The FromMnemonic class used to parse a list of mnemonic words of variable length seems to only report the last error encountered in the list.
• Introducing some failing test case to illustrate this hypothesis and then, if confirmed, fixing it it the way to go.

PR

QA

Context

Once we've defined an SQL database schema for our wallet backend, we'll need a way to perform:

1. Encoding: the conversion of Haskell values into values that can be stored in the database.
2. Decoding: the conversion of database values back into Haskell values.

Decision

When decoding certain values, such as the Wallet state, we may need to trust that the data held within the database is correct.

Acceptance Criteria

1. Values subject to the encoding and decoding process must be able to survive round trips without loss or corruption of data.

Development Plan

PR

• #272

QA

Context

We want to outline a first network layer that abstract away the retrieval of blocks from a chain producer. In the first iteration, this chain producer will be the cardano-http-bridge which serves blocks through a basic simple http API.

Decision

• Implement the necessary HTTP calls to retrieve blocks from the network
• The implementation should satisfy to the following interface:

nextBlocks 
  :: Natural -- | Number of blocks to retrieve
  -> (EpochIndex, SlotNumber) -- | Starting point 
  -> ExceptT ErrGetNextBlocks m [Block]

With m a monad that may be specialized by the implementation. We do not care about rollbacks yet and will introduce them later.

Development Plan

Questions:

1. I would really like to import some property tested slotting arithmetic functions rather than re-implementing them poorly? Is this possible?
2. How are we going to configure number of slots in the epoch?
3. Have we already requested a change to cardano-http-bridge to provide a function to return a range of blocks from the latest epoch?
4. Is servant-client ok for accessing the node api, or just plain http-client?

Plan

• Add some slot arithmetic and comparison functions where necessary
• Number of slots in the epoch hard-coded to 21600.
• Initially model nextBlocks in IO then make it polymorphic over the chain producer backend.
• Unit tests with a mock chain producer.
• Add function to get a range of blocks from the latest epoch, to work around missing endpoint in the cardano-http-bridge API.
• Servant types and client code to model the 3 necessary API endpoints.

PR

QA

• This will be partly covered by unit tests with a mock chain producer.
  • The test cases for nextBlocks are in Cardano.ChainProducer.RustHttpBridgeSpec.
  • The associated mock network layer is in Cardano.ChainProducer.RustHttpBridge.MockNetworkLayer
• The servant-client API has no test coverage (yet). It seems to work, having tried it with informal testing.
• Integration tests with an actual cardano-http-bridge will be possible after #8 is complete. Although executing these tests will require an external dependency (a network to connect to).

Context

Our current implementation has db layer interface defined in https://github.com/input-output-hk/cardano-wallet/blob/master/src/Cardano/Wallet/DB.hs with MVar implementation of this interface defined in https://github.com/input-output-hk/cardano-wallet/blob/master/src/Cardano/Wallet/DB/MVar.hs . This MVar interface gives us ability to have in-memory db system. MVar was implemented because it was simplest thing to test the db layer design/interface. MVar db interface is also very useful because we can use it to run all db actions quick in memory which is useful for implementing tests.

To really have useful db system we have to implement db layer with one that will persist data to disk.

Decision

In a similar way we have added MVar implementation of db layer, add SQLite implementation of db layer. This implementation should use sqlite https://sqlite.org/index.html to persist data to disk. Research available haskell implementations of sqlite and decide which best suits our needs.

Acceptance Criteria

1. DB layer must be implemented with sqlite interface
2. SQLiteSpec module must contain tests for sqlite interface

Development Plan

• Write SQL Schema using Persistent DSL
• Implement DBLayer methods in using Persistent query functions
• Add functions for creating a database connection, and running database migrations.
• Add basic "sanity test" cases while implementing DBLayer methods, while DBLayer is not complete.
• Make the MVarSpec tests run against the Sqlite DBLayer.
• Fix test failures.
• Generate more DBLayer test cases using quickcheck-state-machine.
• Add performance testing to ensure the DBLayer still works with large numbers of wallets/addresses/transactions.
• Use the Sqlite DBLayer by default when running cardano-wallet.
• Command-line parameter to configure database filename.
• (@KtorZ) enable SQLite in integration tests
• (@KtorZ) Fix tx pending & UTxO selectors

PR

• #227
• #247
• #259
• #270
• #276
• #282
• #283
• #288
• #293
• #300
• #345
• #349
• #360
• #367
• #370
• #374
• #377
• #379
• #413

QA

• Automated unit tests are in ./lib/core/test/unit/Cardano/Wallet/DB/SqliteSpec.hs and ./lib/core/test/unit/Cardano/Wallet/DB/StateMachine.hs.
• Run the DBLayer benchmarks with stack bench cardano-wallet-core:db --ba "--output bench.html"
• cardano-wallet now has a --database option. Without that option, it will use the SQLite in-memory database.
• cardano-launcher now has a --state-dir option.
• restoration benchmarks now also use the SQLite implementation instead of the MVar which allows for restoring in rather similar time but, with a much lower memory footprint (<350Mb for the 10% scheme on mainnet).
• Integration tests also run using the SQLite engine.

Context

Daedalus (our main client) works by spawning a corresponding node backend process ensuring the supervision of the node. There are a few steps Daedalus goes through when starting a node as detailed in the diagram below:

Among them, one is of particular interest for us at this stage: the IPC channel. Because it's hard to choose a default port for our application (since on user's machine, another service may already be listening on the port we chose), we need the backend to be able to do a dynamic port selection (ask the OS to assign it an available port) and then, to communicate that port to Daedalus through the IPC channel.

The current implementation of this IPC channel in cardano-sl can be found here and its usage in the codebase here. As a replacement for Cardano.NodeIPC, we want to use the new implementation provided by the cardano-shell here.

Note that, the name NodeIPC can be slightly misleading as it suggest it relates to the cardano node, but it actually refers to node.js here. So, testing of this IPC features can only happen if the launcher process is actually spawned from a node.js process (as an example, here's Daedalus' current implementation).

Decision

We want to make our wallet an IPC server as well and support dynamic port selection (in the case no --wallet-port is passed). That server should only handle two type of messages for now:

• Ping: replying with Pong (to test the connectivity) no need for this.
• QueryPort: replying with ReplyPort and the corresponding port

In light of issue #368, the IPC server logic will be done in cardano-wallet server rather than cardano-wallet-launcher.

Acceptance Criteria

• The launcher cardano-wallet server MUST listen on IPC commands when there's an available handle
• The cardano-wallet server MUST perform dynamic port selection for the wallet API when no port is provided in the CLI
• The cardano-wallet-server MUST use the provided wallet API --port command-line argument if it has been provided.
• The IPC handler MUST handle requests like Ping and QueryPort as described above
• The cardano-wallet server MUST die (and its corresponding sub-processes) when an opened IPC listener is closed.

Development Plan

• Either
  • Investigate Cardano.Shell.NodeIPC code and check whether it can be imported as a library. ⇒ it's not suitable for re-use because testing code is mixed up with message protocol code which is mixed up with the ipc code.
  • Copy&paste cardano-sl NodeIPC implementation ⇒ #388
• Call it DaedalusIPC rather than NodeIPC, because the word "node" is ambiguous and this protocol is unique to Daedalus. ⇒ #388
• In cardano-wallet server, handle the case where NODE_CHANNEL_FD is set. In that case, start a async thread and respond to child_process QueryPort messages (from Daedalus) with the correct port.
• In the DaedalusIPC module, handle the exception when the fd is closed, and gracefully cause the wallet-server process to exit.
• In cardano-wallet server if the --port command line parameter is not given, start the warp server with any available port, and print that port as a log message. ⇒ #387
• Don't copy&paste across the ping/pong messages code because they serve no purpose.
• The integration test will include running a nodejs program that calls child_process.spawn({ command: 'cardano-wallet', arguments: ['server', ...], stdio: 'ipc', ... })
• Ensure the wallet API server is listening on the localhost interface only. In future, an --address argument can be added. ⇒ #387

Last step

• After QueryPort is done, investigate how nodejs can send a socket object to the child_process, so that we can avoid having a TCP server listening on localhost (security risk). This needs to work on both windows and posix.

edit (KtorZ): This decision needs refinement and further investigation, in particular for Windows. So, we agreed to open the discussion again in a while before we implement (or not) TLS.

PR

QA

• A node.js mock IPC client written here
  This basically simulates the interaction with Daedalus, also including illed messages to test parsing errors.

• Corresponding tests for the IPC server using the mock above are located in Cardano.LauncherSpec

• Behavior regarding port selection is also described and tested in Scenario/CLI/Port

Context

This task is part of the Submit transaction story. We need to have our Network Layer extended such that we can post transaction to the blockchain. Cardano-http-bridge has already the endpoint for posting a transaction POST: /:network/txs/signed which needs to be handled in our network layer.

Decision

Extend Network Layer to support posting a transaction via cadano-http-bridge.

Acceptance Criteria

1. The Network Layer must be extended to support POST: /:network/txs/signed endpoint from cardano-http-bridge in order to be able to post transaction to the blockchain.

Development Plan

• I plan on adding something like

type PostSignedTx
    =  Capture "networkName" NetworkName
    :> "txs"
    :> "signed"
    :> ReqBody '[Base64 CBOR] SignedTx
    :> Post '[NoContent] ()

• I plan on getting a Base64 content type working
• I maybe plan on adding a sign :: Tx -> SignedTx function or something
• I plan on getting encodeSignedTx working and roundtrip-tested, while mocking witness.
• I intend to look in cardano-sl for how to create a proper witness.

PR

QA

• CBOR roundtrip tests were added for encode/decodeTxWitness and encode/decodeSignedTx https://github.com/input-output-hk/cardano-wallet/blob/028a5effc62d8783232d5fc5b8446e5ab06812e2/test/unit/Cardano/Wallet/BinarySpec.hs#L127-L152
• We submit txs to the http-bridge and check that we see appropriate error messages in two integration test cases https://github.com/input-output-hk/cardano-wallet/blob/028a5effc62d8783232d5fc5b8446e5ab06812e2/test/integration/Cardano/Wallet/Network/HttpBridgeSpec.hs#L106-L123
• Also have Aeson roundtrip property + goldens for checking the JSON fromat of the SignedTx wrapper (that {"signedTx": <base64>} turns to SignedTx ByteString)

Context

Open a ticket to work on a few technical debts we've accumulated and could fix. The idea is similar to what we did in #94, and have a few bullet points of items that don't necessarily fit into any other milestones but still have some value for the project:

Decision

• (@KtorZ) [#222] Review Buildable instances for primitive types to be more readable & tested
• (@Anviking ) [:eyes: ] Add missing documentation to Cardano.Wallet.Primitive.Types
• (@KtorZ) [#249] Review error message triggered by generate mnemonic in the CLI to display a clean message in read and exit with a failure code (rather than throwing with fail)
• ( @KtorZ ) [Not Sure] Find private keys and redemption certificate corresponding to our local cluster
• ( @KtorZ ) [In Progress] Allow redemption of avvm certificate in the integration scenarios
• (@Anviking ) [#231] Fix documentation in the Data.Quantity module
• ( @KtorZ ) [#223] Get coverage report for integration test scenarios (i.e. run the wallet server in a thread and not in a new process)
• ( @piotr-iohk ) [ #235, #251 ] Add missing integration tests for already implemented API endpoints (post, delete, list, get wallet)
• ( @KtorZ ) [ #249 ] Implement missing endpoints for wallet management

Acceptance Criteria

1. Debts listed above should have been tackled or discarded with justification.

Development Plan

• Pick up a task from above
• Put your github handle @<...> in front to indicate that you're working on it
• Once completed, put a corresponding PR number next to it

PR

• #222
• #223
• #231
• #232
• #235
• #249
• #251

QA

• I believe we addressed all the points written above ^^^
• We still have to port some integration tests related to transactions into our new implementation, but this wasn't really scoped from the above. So likely to be done next week.

Context

We run both cardano-wallet and cardano-explorer on one Ubuntu server. We execute API calls against the wallet to generate new transactions. A few transactions (on daily basis) won't end up in the Blockchain.
Transaction has been generated but stays at:

"confirmations": 0,

We do pay enough fee and we do have enough funds. I used cli-command curl -X GET http://localhost:8090/api/v1/transactions?id={{txid}} | jq . to get the json response which contains the 0 confirmations line.

Steps to Reproduce

1. Executed a new order with the same amount and it'll pass

Expected behavior

The transaction will end up in the Cardano Blockchain

Actual behavior

The confirmations stays at zero.

Resolution Plan

PR

QA

Context

This new wallet implementation will focus on implementing BIP-44's style of addressing and therefore, we can re-use most of the work already done in the legacy implementation and port the following module onto cardano-wallet:

• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/src/Cardano/Wallet/Kernel/Ed25519Bip44.hs
• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/test/unit/Golden/Ed25519Addresses.hs
• https://github.com/input-output-hk/cardano-wallet-legacy/blob/develop/test/unit/Test/Spec/Ed25519Bip44.hs

Decision

1. We should port the above module to cardano-wallet while getting rid of cardano-sl code
2. We should adjust module according to our coding standards and guidelines
3. We may adjust some 'raw' types to be more type-safe (key, indexes ... )

Development Plan

• I intend to mostly copy/paste the content of the Ed25519 module form cardano-wallet
• I'll remove the 'EncryptedSecretKey' and get rid of encrypted key in memory altogether (use of 'ScrubbedBytes' from memory)
• I intend to get inspiration from the wallet prototype and define custom types for indexes and keys to make the meaning of them all a bit clearer and more type-safe
• I intend to also port the relevant property-tests and add a few to get maximum code coverage

PR

QA

• This module define a rather type-safe API when it comes to address derivation, leveraging phantom types on things like indexes and keys (that have a similar internal representation, but a different semantic depending at which stage they're used). This allows for preventing programming mistakes where an address public key would be used instead of a root public key, or when a hardened index would be instead of a soft one (and vice-versa). This can be seen / reviewed from the generated haddock documentation here

• Beside the safety offered by types, we have a few unit tests covering for various things:

  • A set of tests to make sure our hand-written instances are correct see here

  • A set of tests to verify that our main property for sequential derivation holds (the public key of the derived child private key on a soft index is the same as the derived child public key on a soft index, see here

  • An extra set of golden tests which verify that the whole "pipeline", from a root private key down to the encoded address corresponding to an address public key matches those obtained from cardano-cli / Icarus. see here

Context

We'll provide a CLI to interact with the wallet layer from the terminal. The CLI acts as a proxy to the wallet backend server (and therefore, requires the wallet server to be up-and-running) such that every endpoint from the API has an equivalent command in the CLI (which delegates the logic to the API via an HTTP call).

e.g. POST /api/wallets has a corresponding command cardano-wallet wallets create --param1 --param2 --param3

Decision

This first iteration only concerns the following endpoints for which, at least mock API handlers should exists.

• Wallets List | GET /wallets
• Wallets Create/Restore | POST /wallets
• Wallets Get | GET /wallets/{walletId}
• Transactions Create | POST /wallets/{walletId}/transactions

Acceptance Criteria

1. Commands for endpoints listed above must be available in the CLI
2. There should be an extra command to start the underlying web server (e.g. server start)
3. We may rename the cardano-wallet-server to simply cardano-wallet for the CLI has a more general purpose than just being a web-server.
4. Each command should output data in a JSON format, matching the one from the API
5. Sensitive information (like passphrase or mnemonics) must be prompted and not passed as options or arguments

Development Plan

• Define an interface specification using the docopt format.
• Use the docopt Haskell library to parse thedocopt-based specification.
• Use servant-client to define a client interface based on the wallet Servant API.
• For each type in the API: extract out a smart constructor from its FromJSON instance, and reuse this smart constructor when parsing a user-supplied value within the CLI code.
• Reuse record types (such as WalletPostData) defined within the API layer within the CLI layer.
• Share any user-facing error messages between the API and the CLI code.
• Pretty-print the JSON output of calling any API functions.
• Add functions that ask for passphrases: don't accept passphrases on the command line.
• Implement command for transaction endpoint. Because of docopt limitations, we may only allow one ellipse parameter and parse transaction's recipients as <coin value>@<address>, for instance:

  $ cardano-wallet create transaction --target [email protected] --target [email protected]
  

• Write down some manual test procedure

PR

• #133
• #136
• #155
• #161
• #180
• #188
• #189
• #225
• #253
• #254

QA

https://github.com/input-output-hk/cardano-wallet/wiki/Wallet-command-line-interface

Context

SQL like db solution is decided to act as db backend for the wallet.

Decision

SQLite is chosen as SQL db solution. Hence, the schemas for all data to be stored has to be defined. Moreover, the corresponding tables has to be created, and checked that they can be inscribed correctly into the running db process (CREATE TABLE).

Acceptance Criteria

1 SQL schemas for all types
(a) Wallet
(b) WalletMeta
(c) Tx (pending)
(d) The Tx History & corresponding tx meta
(e) SlotId (network tip)
(f) UTxO
(g) AddressPool
to be stored must be defined within sql types
2. The corresponding from point 1 tables must be checked if they can be (CREATE TABLE) to the db and then (SELECTed) from db

Development Plan

• I plan to create SQL schema using persistent lib for : Wallet, WalletMeta, Tx (pending), The Tx History & corresponding tx meta and test them (Ante)
• I plan to create SQL schema using persistent for : SlotId (network tip), UTxO, AddressPool and test them
• I plan to create a module and write few obvious/dummy db selectors that query db
  - this module will consist update/delete/select and other interactions with db
• I plan to create a module and setup code for db migrations
  - this module will consist of future migrations plans and code to execute the migrations
  - would be nice if we could have a way to test migrations - but don't know much about this (Ante)
• I plan to try to create esqualeto version of the above and compare approaches (ask Matthias and rest of the team is this vialable)

(Pawel)
Proposed localization of schema : Cardano.Wallet.DB.Schema

PR

QA

Context

Before we create a proper SQL DB Store, we'd like to finalize the wallet metadata.

In the our current MVar DB layer we're not storing WalletMetadata. We'd like to do that.

The current WalletId is an UUID.

We want the wallet id to be deterministic. We do not want to use (and expose) the root public key directly for security reasons…

Decision

…instead, compute the wallet id using a base16/hex-encoded hash (e.g. Blake2b 128) of root xpub. And — store WalletMetadata in the DB.

Acceptance Criteria

1. WalletMetadata must be stored in the DBLayer
2. WalletId should be redefined as a hex-encoded Blake2b 128-hash of the root public key.

Development Plan

• Replace construction of the wallet id from UUID to an actual hash of the wallet root public key
• Add a property test to verify that any two wallets with the same mnemonics ends up with the same wallet id
• Make sure metadata are stored alongside the wallet state and can be retrieved

PR

QA

• Wallet metadata are now correctly created in the wallet layer when creating a wallet
• UUID has been replaced with an hex-encoded hash of the root public key for the wallet id (this is reflected in the golden tests) and visible through the type of WalletId as well (cf haddock
• I've extended properties for the wallet layer as well to check that the wallet id generation is both deterministic and injective (cf: Cardano.WalletSpec)
• Wallet Metadata are also now stored in the DB Layer (DB layer has been extended to cover for those cases, see here putWalletMeta and readWalletMeta)
• Also added extra tests for the wallet metadata and actually, tx history too in Cardano.Wallet.DB.MVarSpec

Context

As we are starting from scratch an want to add new lean wallet functionality there is a need to structure the code and layers properly. Following the path of previous cardano-wallet we are going to add kernel and API layers to decouple the concerns handle respective errors.

Decision

We are going to add wallet kernel layer that is going to be called from API layer, handle errors, manage persistence and inner workings logic

• we should have separate layer where inner workings of create wallet reside

Development Plan

• I plan to add types/structure/stubs for the first review
• I intend to fill the aforementioned with implementation and corresponding tests

PR

QA

• We have defined a WalletLayer interface that connects the DBLayer and NetworkLayer together.

• There are also a few corresponding property tests ensuring that the create and read functions from the layer works as expected (cf: test/unit/Cardano/WalletLayerSpec.hs).

• More integration tests are coming along with #69

Context

We’re starting fresh in a new repo! We need some basic things to work in it.

We should be able to quickly port over things from the old repo.

Decision

• We must have a .gitignore
• We must have a basic stack setup, with appropriate folder structure (for instance src, test/unit, ...) ⇒ #15
• We should have a config files for hlint and stylish-haskell
• We may have an .editorconfig (This is a proposal in the Coding Standards wiki page)

Development Plan

• I intend to copy the config-files from the old repo
• I might intend to copy the .cabal from the prototype
• I intend to add a pdf export of the formal specifications into a 'specifications' folder
• I intend to add a swagger.yaml file specification document describing the API v2

PR

QA

Context

Since we are starting to shape API for the wallet we want a way to have it tested using integration tests.
This task is about porting the Request module from the cardano-wallet-legacy into new wallet.

For the new wallet's integration tests we want to use only the unsafeRequest approach and give up on using typed-client approach as the first one turned out to be more versatile in terms of producing both positive and negative test scenarios in the past.

Decision

Port all the functions from Request into new wallet that are necessary to submit raw JSON request (unsafeRequest) and get raw response from the request. The response needs to contain response body and response code such that it can be than asserted using test framework's DSL (#55).
Do not port anything related with the typed-client approach.

Acceptance Criteria

1. One must be able to submit a request using raw-JSON as a parameter if needed (unsafeRequest approach)
2. One must be able to get the response from the API (body and response code)

Development Plan

• Copy Request.hs from old codebase
• Keep unsafeRequest and remove servant-client specific code.
• If there is a HttpException, return that to caller in Either type.
• Add a test case to check that unsafeRequest works.
• Call unsafeRequest just request because we don't have typed requests.

PR

QA

The initial test should be run with:

stack test cardano-wallet:integration

This can be considered to be working if the request function can be used for new test scenarios.

Context

We need automated tests to ensure that our wallet meets its performance requirements.

In particular, we need these test cases to detect space leaks, or wrong algorithmic complexity.

Decision

• Come up with test scenarios. For example:
  • receiving blocks from the backend
  • wallet restoration
  • listing transactions
  • input selection
• Make sure benchmarks are run in CI, at least nightly.

Acceptance Criteria

1. The generators must be able to generate sufficiently large data sets along all significant dimensions.
2. Memory usage must be shown for different data set sizes.
3. The benchmarks must be run in CI. They may be run nightly rather than on every push.

Development Plan

Basic

• Use mainnet and testnet data
• The listen polling function (5s interval) will make executing the test more tricky (and testing when it has finished applying blocks) -- make some workaround.
• Time restore operations
• Add benchmark job to CI. Use ghc +RTS -h option to get memory usage statistics.

More interesting

• Change mkWalletLayer so that the address derivation scheme can be swapped out.
• Make an address derivation scheme which "owns" a given proportion of addresses.
• Add more benchmarks with varying address ownership (and hence number of transactions).

Better

Moved to https://github.com/input-output-hk/cardano-wallet/wiki/Blackboard#better-restoration-stress-benchmark

PR

QA

• We now run nightly build restoration benchmarks on buildkite against both mainnet and testnet.

• For each network, we run three benchmarks, playing with different version of 'isOurs' (address discovery)

  • The 'classic' one for sequential wallet. Downside is that we are restoring a wallet from a public arbitrary mnemonic and end up with an empty wallet. Still, we go through the whole chain and process every non-empty block

  • To step-up a bit the game, we also run a 10% isOurs, which recognize, at random, 10% of the address on the chain (using the crc on each address). This allows us to create artificially big(ger) wallets and observe the memory usage for those.

  • Even more, and to observe how the backend scale, we run a 50% isOurs heuristic where half of the chain gets recognized as "ours".

• Builds (with status visible on the README via the NIGHTLY base currently generate a single artifact, which is a heap memory visualization of the benchmark). We don't have any ways of automatically checking the benchmark results at the moment, so this is still something which requires a manual checks. A bullet point to the release checklist has been added to remind ourselves to do so.

Context

As described in the roadmap, we are now creating a new repository and moving away from the legacy code. We have a bunch of things to setup in order to achieve this.

Decision

Create a new repository and import relevants bits from the wiki and the github templates. We also need to re-create the Zenhub board and have proper milestones aligned with our roadmap.

Development Plan

• I intend to rename the old repository to cardano-wallet-legacy
• I intend to create a new fresh repository cardano-wallet
• I intend to import relevant parts of the wiki, and leave out whatever concerns the Byron era
• I intend to import and adjust the github templates (PR, Task, Bug) we've been using
• I intend to setup the Zenhub board to match what's specified in the development process
• I intend to create milestones aligned with our roadmap, give end date to milestones and put corresponding issues in their milestones

PR

QA

• New repository created with same name as before cardano-wallet
• Wiki populated, removed stuff from the Byron Era (see Wiki)
• Added and updated templates for Tasks, Bugs and PRs (removed the "acceptance criteria" section, now conflated with "decisions" and, remove the "retrospective" section; we'll do as part of the iteration planning).
• Adjusted Zenhub board
• Created corresponding milestones (see Milestones) and added corresponding tasks to milestones

Context

A real wallet implementation may need to store more information than we have modelled in the specification so far. For instance, users may wish to know when their pending transactions got confirmed in the blockchain (as opposed to merely that they were confirmed), or what the effect was of a particular transaction on their balance (rather than merely being able to see the current balance).

Decision

Compute and maintain metadata for every transaction tracked by the wallet (transactions send to/from the wallet).

Acceptance Criteria

1. Tx metadata must contain unique identifier (probably tx id)
2. Tx metadata must contain depth (a.k.a "number of confirmations)
3. Tx metadata must contain status: pending vs in-ledger vs invalidated
4. Tx metadata must contain total amount
5. Tx metadata must contain direction: outgoing vs incoming
6. Tx metadata must contain timestamp when transaction was created
7. Tx metadata must contain info about absolute slot & block number the transaction was inserted (from genesis)

Development Plan

PR

QA

Context

We do have a wallet layer which is our main interface to deal with wallet operation and make modification on our various checkpoints.

This layer is missing the ability to submit new pending transactions and sign them!

Decision

Extend the WalletLayer API with a way to submit transaction and keep track of the pending transaction in the underlying checkpoints and database engine. It should make use of the extended interface for the network layer and the coin selection (or at least, a mock of it).

Note that, to keep enough flexibility and allow for easily implementing already existing requests from various users, we do want to keep the input selection, transaction signing and transaction submission three separate actions. API handlers will eventually put them all together.

Acceptance Criteria

1. Transaction submission (of an already signed transaction)must be supported by the wallet layer
2. Transaction signing must be supported by the wallet layer
3. Coin selection must be supported by the wallet layer

Development Plan

• Port things from cardano-sl and experiment
• Simplify drastically
• Connect with the WalletLayer
• Re-add features that "were lost"
• Test by comparing implementation with cardano-sl

PR

QA

• This one was quite ... huge in the end. Many things have been done, starting from extending the wallet layer (see haddock) with three functions:

  • createUnsignedTx: run a coin selection across the wallet to build a transaction from available funds
  • signTx: produce witnesses for a given coin selection, requires a private key to have been stored
  • submitTx: broadcast a transaction to the network

• At the API-level, a transaction handler has been implemented to perform the three steps above in sequence. We may consider later to introduce new endpoints to perform only some of those steps.

• Regarding the transaction itself, there are a few things which are network-specific and depends on the binary representation of a transaction. We have therefore introduced an abstraction 'TransactionLayer' which captures this specificity. Golden tests with cardano-sl have been added here and make sure that we do generate and encode transactions exactly like cardano-sl implementation would do it.

• Note that, there's currently no integration tests which verifying the end-to-end process of creating a transaction from an API call and watching it being picked up by the wallet. My two cents on this: this isn't trivial to do as it requires us to be able to make transactions, and therefore, to redeem funds from a faucet address first! I'd therefore suggest to release the feature as "experimental" for now in the next pre-release, and have a dedicated milestone about doing this. Regarding priorities, since transactions are going to have a different format in the Rust node and, a slightly different logic regarding witnesses, it'd be better to perform integration tests directly with the rust node instead of loosing a week or two to enable this with the http bridge.

Context

The task being part of Receive And Process Blocks user story.

In the new architecture we are pursuing pull mechanism for getting blocks. It means the wallet is responsible for downloading the blocks. In order to do it the wallet has to poll for the current header tip of the block, and then download the corresponding block

Decision

We will add ticking mechanism that polls the cardano-http-bridge periodically. First, at 5 seconds interval in order to make sure that every slot is queried and hence backward downloading logic is not needed. The logic will bog down to following loop:
(a) polling the current header tip of the block
(b) using the obtained header (if it is different from the one of the previous download)
to download the block
(c) delay thread for 5 seconds

Development Plan

• I plan to add ticking function that gets blocks and apply action if not yet done before
• The reasonable test is to be added
• When wallet primitive types are in place, I make sure I align

PR

QA