To simplify basic things like getting a storage miner peerid, something like:

act := apiactor.StorageMiner(api) // Could also take options like tipset
peerid := act.GetPeerID()

We may also want a variant, which can commit messages to chain, but that can be done later.

with a daemon running, some commands work, chain head doesnt:

why@WhyNet ~/g/s/g/f/go-lotus> ./lotus wallet list
t1hzvnuhscaz7vgekajur7ipqrqrwgmdv56ne3nna
t3rv34lpcpxoqx6scl767zzdnsuiipfoej3owelsmluyx4z6cz5eenxdtxhrrttmz5e7quwshwqb4v7nvtsejq
why@WhyNet ~/g/s/g/f/go-lotus> ./lotus chain head
2019/07/17 22:39:14 RPC client error: unknown address network

#61 (comment)

(We also should store api token with limited permissions)

See if existing jsonrpc clients actually work with our lib

We should construct a set of synthetic benchmarks for chain processing.

Some initial ones i'd like to see:

1. process a 100,000 block chain, with no messages
2. process a 100,000 block chain, with N={1,10,100,1000,10000} messages per block

• messages just moving money to random accounts
• different benchmarks for messages signed with BLS vs secp256k1 vs some mixture

3. process a chain with N={1,10,100,1000} sector commitments messages per block
4. process a chain with N={1,10,100,1000} payment channel operations (opens/updates/closes)

More later for sure, but this is a good start.

The storage miner should be a separate process that interacts with a 'full node' process.

The storage miner is responsible for:

• accepting deals
  • managing the entire deal flow
• writing the data into the sector base
• sealing sectors (might be handled by sector base? unclear where the boundaries are)
  • submitting sealed sectors to the chain
• computing proofs of space time and submitting them periodically

In order to make this all work, we need to expose a number of APIs:

• an API that allows us to read miner state
  • need to be able to check sector set, proving set, proving period times, etc to ensure it matches whats going on locally.
• api to submit messages
  • should signing these messages happen over the api? the storage miner process will have its own keys as well
• api to wait for messages to land in the chain (this could be done a few different ways)
• probably more

• Implement VM Invoke interface
  • Implement Actors:
    • InitActor
    • StorageMarket
    • Miner
    • Payment Channel
    • Multisig
    • AccountActor
• Wallet
  • Key generation
  • Key export
  • Hardware wallet integration
• storage miner implementation
  • requires integrating rust proofs code
• write a program to generate genesis blocks from some template file (initial accounts from given public keys, see gengen in go-filecoin)
• implement retrieval protocols (https://github.com/filecoin-project/specs/blob/master/retrieval-market.md)
• implement "AMT" for sector set storage (filecoin-project/specs#116)

Should be easy with libp2p eventbus

When referencing messages by their CID, we need to take into account that signatures are stripped from BLS signed messages before inclusion into a block. So if we take the Cid of a BLS 'SignedMessage' (where the hash includes the message) then someone looking into the block for that message will not be able to find it. We need to make sure that the Cid of a BLS signed message is always just the Cid() of the unsigned 'Message'.

thoughts @Kubuxu @magik6k @dignifiedquire ?

I wrote gengen for go-filecoin. We probably want to do the same thing here, just with the spec compliant format.

We need a more optimized datastructure for storing sector metadata in the state tree. The HAMT (probably) won't cut it in the long term.

Research has been done in this direction here: filecoin-project/specs#116

We should implement the described datastructure and start using it.

suspect that this is resolved in #93, but filing an issue just in case

Currently when the client disconnects the behaviour isn't clearly defined.

When the client loses connection, we should block all calls and try to reconnect in background.

It will allow us to understand why it takes so long (if it does) and where is the performance going.

This will be a change in the codegen lib, but we need to make sure it happens

i'm working on moving the block miner to be spec compliant. Part of this involves several signatures with the miners worker key.

This means that the process creating blocks on behalf of a given storage miner needs access to that storage miners key.

I think the most straightforward way of doing this would be to just have the full node keep the key for the storage miner, instead of the current situation where the storage-miner has its own wallet.

thoughts @magik6k @Kubuxu ?

Theres a few new test cases we should add to the syncer. I'll just descibe them here briefly, using a notation of 'we want to sync from A to B, and the chain progresses to the right, and [ ... ] represents 'any number of blocks'

[ A ] - [ ... ] - [ B ]

[ ] - [ ... ] - [ A ] - [ B ]

[ ] - [ ... ] - [ A ] - [ ... ] - [ B ]

[ ] - [ ...] - [ ] - [ ] - [ A ]
                |
               [ ] - [ ] - [ ] - [ B ]

definitely more to come after this, but if we make testing scenarios like this easy, then we can get some pretty good coverage.

If the current state of the chain says the nonce is too low, we should error out immediately (failing out the call the MpoolPush)

We should be doing some amount of block validation before rebroadcasting blocks over pubsub, not the full validation, but some structural validation.

This is described in the block propagation spec: https://github.com/filecoin-project/specs/blob/master/data-propagation.md#block-propagation

ref: https://github.com/filecoin-project/specs/blob/master/data-propagation.md

Blocks are persisted right now, but the latest head is not, so when i restart my chain everything is brand new (unless i sync from some other peer)

Implement the multisig wallet.

This involves first writing and testing the actor (docs here ). Then, some API and CLI interfaces so the user can interact with it and use it.

Users should be able to use their hardware wallet to sign transactions. We should investigate how much effort it would be to add support for this directly in lotus (given some past experiments, I think its pretty straightforward).

right now the block reward is a fixed number, need to change this to the correct values from the spec.

Reference the actors doc and the payments doc

First implement then actor, and add some tests. Then move on to API and CLI interface to the payment channels. We want to encourage people to use payment channels whenever possible for paying others, so making these APIs nice and usable is important.

quick sketch of cli thoughts:

lotus paych create <target> <total amount>
lotus paych voucher <channel> <amount> # sequence IDs could be tracked internally
lotus paych redeem <voucher>
lotus paych status <channel>
lotus paych list # list open channels to and from us

Currently inokation interface is:

Send(to address.Address, method uint64, value types.BigInt, params []byte) ([]byte, uint8, error)

It is hard to differentiate which errors should be reported as errors and which should be return codes.

There are two ideas I have, either we change the interface to:

Send(...) ([]byte, actorError)
type aError struct {
   msg string
   err error
   frame xerrors.Frame
   retCode uint8
   fatal bool
}
type actorError *aError

func NewFatalErr(msg string, err error) actorError {...}
fund NewRetErr(msg string, code uint8, err error) actorError {...}

// Fatal marks an error that should abort execution of message
func (actorError) IsFatal() bool {
   return actorError.fatal
}
func (actorError) RetCode() uint8 {
   return actorError.retCode
}
func IsFatal(err actorError) bool {
    return err != nil && err.IsFatal()
}
func RetCode(err actorError) uint8 {
    if err == nil {
        return 0
    } else { return err.RetCode() }
}

And create wrappers around all apis to return actorError.
Then handling of api call would be:

// If we want to recover
result, err := vmctx.Something()
if IsFatal(err) {
  return nil, err // or: nil, Wrap(err, "optional msg")
}
if RetCode(err) != 0 {
  // Recover
}

// If we don't want to try to recover
// If we want to recover
result, err := vmctx.Something()
if err != nil {
  return nil, err// or: nil, Wrap(err, "optional msg")
}

We're currently using FCS for serialization. The protocol working group has since decided that that wasnt the best idea, and now we're just using tuple serialized cbor-ipld everywhere. The net change is to stop using a custom CID, and remove the tag prefix on all the objects.

Additionally all the other objects in the system need to be serialized this way as well, including actor state, and parameters. We should even investigate using this for the HAMT's internal serialization.

In order to make testing easier, and also to generally improve usability, lotus should make it easy to import and export chains to a car file. Additionally, lotus should be able to verify a car file in place (without importing).

The immediate value here is that it makes cross-implementation testing much easier to perform, simply run lotus for a while, do a bunch of things, export the chain, and make sure another implementation validates it.

$ lotus chain import chain.car
$ lotus chain export chain.car
$ lotus chain verify chain.car

In order to program around certain things happening (or not happening) on chain, we need a robust api that makes this easy to think about from the point of view of the person writing the logic.

Title says it all. obviously the daemon shouldnt crash, but we could probably detect the connection close and exit.

We should be able to run make bench to run all the benchmarks and produce a common output format. Something like:

{
  "commit": {
    "hash": "12fde71ba723bce7",
    "parent": "dbb8912efa73b3c3",
    "date": "03-07-2019 4:17 UTC"
  },
  "tags": ["something"],
  "submitter": "whyrusleeping",
  "runDate": "Tue Jul 30 11:55:58 PDT 2019",
  "system": {
    "cpu": "i7-9900k",
    "os": "linux 4.21"
  },
  "benchmarks": {
    "chain-process-10000-m10": {
        "timeperop": {"value": 12501492, "type":"ns", "stdev": 12662},
        "allocsperop": {"value": 125, "type": "count"},
        "bytesperop": {"value": 150000, "type": "bytes"},
    },
    "serialize-block": {
        "timeperop": {"value": 129853, "type": "ns"},
        "allocsperop": {"value": 4, "type": "count"},
        "bytesperop": {"value": 126520000, "type":"bytes"},
    },
    .... many more ...
}

Once its easy to run benchmarks and produce nice output like this, we can write a dumb little web server that collects these (maybe with some auth?) and displays them on a graph. The engine working group can probably help us out there.

Followup from #37, the actors code is done, now we just need a CLI interface.

Copied from the sketch in the previous issue:

lotus paych create <target> <total amount>
lotus paych voucher <channel> <amount> # sequence IDs could be tracked internally
lotus paych redeem <voucher>
lotus paych status <channel>
lotus paych list # list open channels to and from us

Got this while playing around with pond

runtime error: index out of range

goroutine 1523 [running]:
github.com/filecoin-project/go-lotus/chain.(*ChainStore).NearestCommonAncestor(0xc00021a910, 0xc009602e80, 0xc0095b3dc0, 0x0, 0x0, 0x0)
	/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/chain.go:428 +0x10d
github.com/filecoin-project/go-lotus/chain.(*Syncer).selectHead(0xc0003cd740, 0xc0003ee6c0, 0xc009607d70, 0x1, 0x1)
	/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:416 +0x257
github.com/filecoin-project/go-lotus/chain.(*Syncer).SyncBootstrap(0xc0003cd740)
	/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:195 +0x134
github.com/filecoin-project/go-lotus/chain.(*Syncer).InformNewHead.func1(0xc0003cd740, 0xc0095b3d40)
	/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:154 +0x13b
created by github.com/filecoin-project/go-lotus/chain.(*Syncer).InformNewHead
	/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:148 +0x143

To make starting test networks, it would be really nice to have a single command that creates a new genesis block filled with everything needed to initialize a network, starts mining on that genesis block (using keys and actors generated in that block) and then writes out both that genesis block and the bootstrap address of its libp2p node for others to connect to.

This was mentioned on Slack, opening an issue so this doesn't get lost

This is a meta issue looking at what is still needed for complete deal flow:

• Before deal
  • Storage market
    • Spec
    • Dependencies
      • ?
    • Query
• Deal flow
  • Specs
  • [client] Setup payment channels
    • Spec - ?
    • Do we send multiple vouchers so the miner gets paid over time?
    • Depends on #119
  • [client] Send proposal
    • [client] Persist in datastore
  • [miner] Process proposal
    • [miner] Verify payment
    • [miner] Persist in datastore
    • [miner] Send resoponses
    • [miner] Fetch the data
      • [miner] Bitswap
      • [miner] Graphsync
    • [miner] Add to sector
      • [miner] Send PieceInclusionProof to client
    • [miner] Seal sector
      • That's not what AddPiece does, is it?
      • [miner] Put commit message on chain
      • [miner] Send commit message cid to client
• Duration of the deal
  • [miner] Generate PoS
    • Do we send / make these queriable for the client?
    • Where is the spec?
  • Deal checks
  • ArbitrateDeal
  • Basic retrieval protocol
• After the deal / proving period
  • [miner] Post PoSt
    • Apply Late submission penalty
    • Reported storage faults
    • Doesn't this also happen periodically while the deal is up?
  • [client] Check if miner submitted PoSt, and that sectors we care about are included
• After the deal
  • Cleanup datastore entries
• Anything else? (feel free to edit adding missing things)

The api needs a few things from a security standpoint:

• CORS/XSS protection
  • as we all know, websites can tell your web browser to make arbitrary requests to localhost addresses. This seems kinda dumb, but its what we have to deal with. Protect against this
• API authentication
  • We should have a story for how to have a password authenticated API, these discussions never went very far in IPFS-land, but i think this is more critical to have here.

Implement the retrieval protocol.

This is a very basic "send the miner money and hope they are kind enough to actually send back the data" sort of thing. We will also need some basic API/CLI interface to it, i sketched some out in the linked doc a while ago, but its likely not ideal.

Running this command creates the repo immediately, but if it fails, leaves it in place. We should clean up after ourselves.

We need to charge gas for the execution of messages. This has to be something that ends up in the spec, but good values will have to be driven by development.

In a recent go-filecoin PR, @laser extracted a bunch of the sector builder binding code into a separate package. Now we need to pull that into a separate repo so we can import it and use it.

Additionally, it will be generally useful to be able to easily write standalone tools that utilize that code.

Gas charging of them is a bit weird right now and IDK what should happen with them in general.

What I mean by NOOP is not invoking a method nor transferring funds.

We should have benchmarks for the serializations of our types, include the basic ones like block and message, and the more complex ones like the HAMT.

This test is time based and occasionally fails on CI. We should integrate the eventbus at some point to consume sync events and use that to signal when the test should move on

# base level necessary things
lotus wallet sign-msg HEXENCODEDMESSAGE
lotus mpool push HEXENCODEDSIGNEDMESSAGE
lotus mpool pending

# calls method on an actor, does not send a message, really only for 'read
# only' things as a way to query chain state
# NOTE: ethereum has an explicit "estimate gas" api. We could also have that, but having this call return a receipt with the gas it would have used might be a nice api
lotus ??? call ADDRESS METHOD PARAMS...

# sends a message to the network invoking the given method with params on the
# specified actor. under the hood, this creates, signs, and pushes a message to
# the network
# To consider: should this block until its included in the chain? (at least multiple minutes)
lotus ??? invoke ADDRESS METHOD PARAMS...

# TODO: wallet commands to create and manage keys (take hardware wallets into account)
# go-filecoin wallet code (by dig) was inspired by ethereum, we should
# investigate, its likely good to use
lotus wallet ??
lotus wallet list
lotus wallet balance ADDRESS

lotus chain head
lotus chain getactor ADDRRESS # shows all actor info, like balance, code cid, state root, ID address and resolved normal address
lotus chain getblock <by height | by hash>
lotus chain getmsg <hash>
lotus chain pushblock <serialized block + cid list> # for a miner to submit a block they made

lotus node status # maybe too general?
# alternatively... just get the specific thing we want
lotus node issyncing
lotus node gasprice

lotus swarm # libp2p funtimes
lotus config # you know the drill

# TODO: Mining api!
# if this is a separate process, it needs an api, if its not, it needs api entries here...
# we want to be able to support multiple different miner actors in a single process
# do we separate block production from proving? if so, how?
lotus-miner status
lotus-miner set-price <miner> <price> # assuming theres only one discriminator for whether or not to accept deals
lotus-miner list-miners
????

Ethereum API for reference: https://github.com/ethereum/wiki/wiki/JSON-RPC
Bitcoin core api: https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_Calls_list
Zcash api docs: https://zcash.readthedocs.io/en/latest/rtd_pages/payment_api.html (they extended bitcoins apis a bit to add stuff for sheilded addresses, including a sort of 'background jobs' api for snark proving)