• Deprecate request service and channel use

• Rename and repackage

• Remove side client and client keys from config

• Prepare use with service interface

• Prepare use with db interface

• Remove regtest stuff from main

re think timing delays on:

• missing / no commitments

• already attested

• other...

The Commitment Model will replace the latest Ocean hash being used so far for attestations. This will incorporate the hashes from all clients into a merle tree. Template:

type Commitment struct {
 commitments type map < position , hash >
 root type hash
 updated type bool
}
GetRoot() // calculate and update merkle root
GetHashForPosition(position) // get hash at position
UpdateHash(position, hash) // update commitments and trigger root update

Changes required:

• Commitment model

• Merkle tree util and testing

• Attestation model to move from Hash to Commitment

• AttestService to request/store/use Commitment instead of hash

• Server to store latest Commitment not Hash per client

• Commitment proofs

• ClientDetails model

• Tool to receive client key, assign new position, generate new auth token and update db

https://godoc.org/github.com/satori/go.uuid

Max fee not enough to confirm a transaction and no way to topup

commerceblock/mainstay-mvc#52

example:

• registered clients at positions 0, 1, 2
• only client 0 and 2 have sent commitments

Mainstay will fail for missing a commitment in position 1. How do we handle this?

• multisig socket communication
• attestation client to generate multisig addresses and tweaking
• client tool to manage hashes, redeem script and send signatures
• signature collection, tx signing and send

• tweak extended key and test

• changes to client and service and test

• add error handling for tweaking error. test tweaking and constructor errors.

• config for chain code and changes to client

• Modify verification tool and any other tools (?)

New Listener class to be used by AttestService and replace the call to GetNextAttestationHash(). The service should get the latest hash by querying the Listener.

AttestListener, AttestListenerFake with single method GetNextHash() for now

AttestListenerFake will use dummy sidechainclientFake for unit-tests and AttestListener can do what
GetNextAttestationHash() is doing for now.

Update unit-tests and verify same behaviour

Reference file models/request.go

• Create generic struct for request GET method
• Create generic struct for request POST method

Fees are currently estimated without the scriptSig, which accounts for 3/4 of the transaction size.

Thus our current limits of minimum 10sat/B and maximum 100sat/B actually correspond to 2.5sat/B and 25satB/ respectively.

It is not a real problem at the moment as fee limits can be set at will and transaction size is going to be pretty much constant throughout, but still something that should be taken into account.

Store various information on the attestation bitcoin transaction for display through api.

script to send block hash commitments to mainstay server

There should be no need to rescan when importing a newly created tweaked address, since it is inefficient and might take minutes. Currently this is not supported by btcd.

1. AttestServer

• receive hash POST request from requestapi via request channel
• verify clientId, store hash, reply ACK via response channel
• nextHash requests
• latestAttestation requests

2. AttestService

• Send hash requests to AttestServer
• Redirect commitment requests to AttestServer

Staychain changes:

• chain verifier to verify Proof and Tweaking given proof from API. Ocean client used only for block hash validation
• Extend confirmation tool config options (i.e. client position, api, ...)
• Guideline
• forward / backward mode in confirmation tool
• change chainfetcher to have a backward mode

Steps towards making attestation service extra secure.

• wallet failure handling (attestservice:stateInitWalletFailure())
• deprecate using lastunspent / unconfirmed from mempool
• store latest commitment on ipfs

Currently sending full transaction bytes

Send only pre-image (one per input) to signers

Any excess inputs are signed with topup key

More more than one input possibly use:
[size1] [pre-image1] [size2] [pre-image2]

similar to what we do with signatures on the other end.

• Extend config to include top-up addr and top-up flag. Move staychain config in one category:

"staychain" : {
 "regtest" : ,
 "initTx" : ,
 "initScript" : ,
 "topupAddr" : ,
 "topupScript" : ,
}

• Attest client changes
  Examine cases we've been assuming single vin so far
  Do creation, signing for multiple inputs
  Unit-tests

• Attest service changes
  New DO_TOPUP state
  Fetch topup unspent and create attestation
  Fetch multiple sigs from clients
  Unit-tests

• Unit-test top-up scenario paying to new address

• Testing with txsigningtool (need additional -topupPk)

• Raise warnings when funds below some value && raise error when below maxFee allowed

1. Extend requestapi

• New POST request commitment/send/{template} for clients to send commitments to service
  e.g. template
  {
  clientId: ""
  hash: ""
  height: ""
  }
  This request will be passed to requests channel, reach AttestListener, who will reply true/false depending on whether clientId was valid.
• New GET request server/verify/{hash} to request whether a hash was attested
  This request will be passed to requests channel, reach AttestServer, who will reply true/false depending on whether hash has been attested yet. (see "/block/{blockId}")

2. Refactoring

• In models, move Response/Request/Channel to separate files.
• Create typed names for Request, e.g. type VerifyHashRequest = "VerifyHash". Replace hardcoded names in request routes.go with typed names. Replace hardcoded use of request in attest server.go with typed names. Request.name should be this typed name. Remove Request struct from Response struct.
• Rename route handles to local, i.e. Block -> handleBlockRequest
• Rename routes to /api/commitment/send, /api/server/verify/, etc..
• Any changes to unit-test

Replace any remaining "fmt" calls with logger calls. Consider all info/warn/error levels.

commerceblock/mainstay-mvc#49

Rebuild wallet from initial tx/pk
Similar concept to confirmation tool, but this also imports all derived keys to the wallet as well
Proof that in case of failure attestations can continue just with initial pk

Examples

• client sign up on mainstay website

Use cases

• Plain commitment and verification

• Doc history (4548d4d)

• Ocean commitment and verification

• Other

Is using the fee estimation for 21.co ok long term? The company has migrated to earn.com so I doubt its a long term solution.

Major changes related to HSM integration

We need separate permissions for the API and the mainstay daemon. I created this https://github.com/commerceblock/mainstay/blob/develop/scripts/db-init.js months back but I think we should integrate it now to the mongo deployment.

Basically two separate roles. Service role should only be used by the mainstay daemon. The API role should be used by the node web backend. Nobody else should have access or we could introduce a third read only role.

Compose files should be updated with the new users.

error is un-exported and thus not encoded

Add special handling when Response returns with error

cmd/clientcommitmenttool/clientcommitmenttool.go

• Attestation schedule
  Different schedule for Attestations per hour via config
  Different schedule for confirmation wait time before fee bumping via config
  Different schedule for other state wait times, i.e. waiting for signatures

• Extend unconfirmed handling
  If no unspent, do additional search in dB and check blockchain
  If not in mempool, do additional search in dB and check blockchain

• Bump fee handling
  Fee struct with last fee, fee, bumpFee, upper fee limit
  GetFee(), BumpFee() methods in client
  do NewFee() on each new attestation

Create testing function

• HandleBlock
• HandleBestBlock
• HandleBestBlockHeight
• HandleCommitmentSend
• HandleIndex
• HandleLatestAttestation
• HandleServerVerify
• HandleTransaction

Use current time on pre send store and tx time on confirmed store.

Extend test to use 2-of-3 P2SH
Extend test to have a slice of configurations
Use a single rpc, but an AttestClient instance for each
Extend attestsigner_fake to have slice of configs

Mainstay service subscribes to signers to get signatures. A slight delay from the signer causes mainstay to resubscribe possibly ending with 2 sigs in the queue. This leads to a cycle of fail-success attestations.

This was not evident before as in production we wait for 1 minute for signatures, which is never a problem. In local testing it was also never a problem as sigs were propagated in under 10 sec (the test waiting time). I noticed this because HSMs take their time.

There was a similar problem in the federation zmq connections. Solution: exhaust subscriber queue.

"signrawtransaction was removed in v0.18"

Since signrawtransaction withwallet/withkey not supported in btcd and most of signing will not be done by a node anyway, deprecate and replace with dummy signing for testing.

Split tweak into 8 4-byte fragments that define the HD path.

Derive tweaked key using this derivation path.

Using DNS names for zmq subscribe means IP of publisher might change.

Do periodic / on-failure resubscribe.

• mainstay

• txsigningtool

• hsm interface

• Db config

• Db insert interface

• Db find interface

• Full Db integration to server/service

• Db integration testing

1B. Listener

Simple demo of listener - client communication at cmd/listenerdemo.go

cmd/clientcommitmenttool/clientcommitmenttool.go

• connect to sidechain
• new dealerZmq to connect to listener router
• send latest hash and wait for confirmation

AttestListener

• new routerZmq to listen to client for received hash
• store latest hash in the struct
• reply latest hash to incoming request
• send confirmation to client when attestation is confirmed

AttestService

• Send message via requestChannel when requiring new block hash and wait for reply on responseChannel
• Send confirmation via confirmationChannel when attestation is confirmed on the bitcoin blockchain

Listener main should run in a separate goroutine than attestservice. Functionality:

• listen for received hash
• return GetNextHash() if there is a request from attestservice (requestChannel, responseChannel)
• reply to clients with confirmation if there is an attestation confirmation from attestservice (confirmationChannel)

for {
    select {
        case conf := <-confirmationChannel:
            sendConfirmation()
        case req := <-requestChannel:
            responseChannel <- GetNextHash()
        default:
            listen()
    }
}

zmq background
http://api.zeromq.org/2-1:zmq-socket
https://github.com/pebbe/zmq4/blob/master/examples/rtdealer.go
http://zguide.zeromq.org/php:chapter3#toc19