status-im / nimbus-eth2 Goto Github PK

Nim implementation of the Ethereum Beacon Chain

License: Other

Nim 95.05% Shell 1.93% Makefile 0.65% Python 0.16% HTML 0.11% C 1.71% Jupyter Notebook 0.23% Objective-C 0.04% Nix 0.03% Assembly 0.08%

ethereum nim beacon-chain serenity nimbus eth2-clients nimbus-eth2 eth2 proof-of-stake hacktoberfest

nimbus-eth2's Introduction

Nimbus Eth2 (Beacon Chain)

Nimbus-eth2 is an extremely efficient consensus layer (eth2) client implementation. While it's optimised for embedded systems and resource-restricted devices -- including Raspberry Pis, its low resource usage also makes it an excellent choice for any server or desktop (where it simply takes up fewer resources).

Documentation
Related projects
Donations
Branch guide
Developer resources
Tooling and utilities
For researchers
License

Documentation

You can find the information you need to run a beacon node and operate as a validator in The Book.

The Quickstart in particular will help you quickly connect to either mainnet or the Prater testnet.

Quickly test your tooling against Nimbus

The Nimbus REST api is now available from:

Note that right now these are very much unstable testing instances. They may be unresponsive at times - so please do not rely on them for validating. We may also disable them at any time.

Migrate from another client

This guide will take you through the basics of how to migrate to Nimbus from another client. See here for advanced options.

Related projects

status-im/nimbus-eth1: Nimbus for Ethereum 1
ethereum/consensus-specs: Consensus specification that this project implements

You can check where the beacon chain fits in the Ethereum ecosystem in our Two-Point-Oh series: https://our.status.im/tag/two-point-oh/

Donations

If you'd like to contribute to Nimbus development, our donation address is 0x70E47C843E0F6ab0991A3189c28F2957eb6d3842

Branch guide

stable - latest stable release - this branch is recommended for most users
testing - pre-release branch with features and bugfixes slated for the next stable release - this branch is suitable for use on testnets and for adventurous users that want to live on the edge.
unstable - main development branch against which PR's are merged - if you want to contribute to Nimbus, start here.

Developer resources

To get started with developing Nimbus itself, see the developer handbook.

Tooling and utilities

We provide several tools to interact with ETH2 and the data in the beacon chain:

ncli - command line tool with pretty printers, SSZ decoders, state transition helpers to interact with Eth2 data structures and functions
ncli_db - command line tool to perform surgery on the Nimbus sqlite database
multinet - a set of scripts to build and run several Eth2 clients locally

For researchers

Block simulation

The block simulator can quickly run the Beacon chain state transition function in isolation. The simulation runs without networking and without slot time delays.

# build and run the block simulator, then display its help ("-d:release" speeds it
# up substantially, allowing the simulation of longer runs in reasonable time)
make NIMFLAGS="-d:release" block_sim
build/block_sim --help

Local network simulation

The local network simulation will create a full peer-to-peer network of beacon nodes and validators on a single machine, and run the beacon chain in real time. Parameters such as shard, validator counts, and data folders can be set in as environment variables before launching the simulation.

# Clear data files from your last run and start the simulation with a new genesis block:
make VALIDATORS=192 NUM_NODES=6 USER_NODES=1 local-testnet-minimal

# In another terminal, get a shell with the right environment variables set:
./env.sh bash

# In the above example, the network is prepared for 7 beacon nodes but one of
# them is not started by default (`USER_NODES`) - this is useful to test
# catching up to the consensus. The following command will start the missing node.
./tests/simulation/run_node.sh 0 # (or the index (0-based) of the missing node)

# Running a separate node allows you to test sync as well as see what the action
# looks like from a single nodes' perspective.

By default, validators will be split in half between beacon node and validator client processes (50/50), communicating through the common validator API (for example with 192 validators and 6 nodes you will roughly end up with 6 beacon node and 6 validator client processes, where each of them will handle 16 validators), but if you don't want to use external validator clients and instead want to have all the validators handled by the beacon nodes you may use USE_VC=0 as an additional argument to make local-testnet-minimal.

Alternatively, fire up our experimental Vagrant instance with Nim pre-installed and give us your feedback about the process!

Visualising simulation metrics

The generic instructions from the Nimbus repo apply here as well.

Specific steps:

# This will generate the Prometheus config on the fly, based on the number of nodes:
make REMOTE_VALIDATORS_COUNT=192 NUM_NODES=6 USER_NODES=0 local-testnet-minimal

# In another terminal tab, after the sim started:
cd tests/simulation/prometheus
prometheus

The dashboard you need to import in Grafana is grafana/beacon_nodes_Grafana_dashboard.json.

CI setup

Local testnets run for 4 epochs each, to test finalization. That happens only on Jenkins Linux hosts, and their logs are available for download as artifacts, from the job's page. Don't expect these artifacts to be kept more than a day after the corresponding branch is deleted.

License

Licensed and distributed under either of

MIT license: LICENSE-MIT or https://opensource.org/licenses/MIT

Apache License, Version 2.0: LICENSE-APACHEv2 or https://www.apache.org/licenses/LICENSE-2.0

at your option. These files may not be copied, modified, or distributed except according to those terms.

nimbus-eth2's People

Contributors

Stargazers

Watchers

Forkers

djrtwo hankballew sriharikapu gravityblast eos21 binaryflesh nimblenode jangko araq prestonvanloon vers10ne jrhea metaver5o cryptomental acolytec3 jgcarv zedeus protolambda shyran-systems d-nice syuan100 unixpi vbuterin mul1sh ralexstokes tumtum23 rista404 asymmetric kidkal q9f jclapis actuallymentor zah feliciaxml mpetrunic chrishobcroft py-foundation-group-sa bauerj beroburny metacatledger alorehermit tomlushikong bernardoaraujor felixboehm wiltonlazary 1two3code zachinquarantine etan-status scottypoi petarkirov skalarfeilds79 eccentricexit fredriksvantes kimtrai2020 seanbrennan95531 muddlebee lbly emilivanichkovv freedomshare cxz chfast sbnair yyoncho hool1475 equitytrustbankxfether0xlba0ethplorer paul0627 jmcpheron harpocryptes tokensdl mcdee isabella232 michaelsproul woodpile37 kayla-henrie ace7132323 arnetheduck metacraft-labs daleboy asnfaderopo spook-123 pinkdiamond1 lilsunny243 sitedata tdahar antid2ta 22388o dynm shiti benv666 watcharamonthon goleft4e shalokshalom julian-st golffcoin thomaseth2 william3johnson inferno-inc julius1999-md-gmail-com dapplion ronsherfey

nimbus-eth2's Issues

[Simulation] Master node can be disconnected due to "BreachOfProtocol" / Future still in progress

As of https://github.com/status-im/nim-beacon-chain/tree/e4c10a31c9ab147ff0cffc3c8b2d120ddae69806

Errors:

INF 2019-03-26 09:34:28+01:00 Block resolved                             tid=4794 blck="(slot: 15, previous_block_root: \"1270EF65\", state_root: \"D9DEA76F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad15fd9f\")" blockRoot=EC2C20F8 node=4
INF 2019-03-26 09:34:28+01:00 Forward sync imported blocks               tid=4794 badBlocks=1 bodies=14 goodBlocks=13 headers=14 roots=14 node=4
INF 2019-03-26 09:34:28+01:00 Preparing for fork choice                  tid=4794 connectedPeers=0 node=4
DBG 2019-03-26 09:34:29+01:00 Replaying state transitions                tid=4794 ancestors=1 prevStateSlot=0 stateRoot=246EF52C stateSlot=15 node=4
INF 2019-03-26 09:34:29+01:00 Storing state                              tid=4794 stateRoot=246EF52C stateSlot=0 node=4
DBG 2019-03-26 09:34:29+01:00 No head update this time                   tid=4794 headBlockRoot=DB12D744 headBlockSlot=12 node=4
NOT 2019-03-26 09:34:29+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=4794 expectedEpoch=4 expectedSlot=38 fromNow=3 headSlot=12 node=4
INF 2019-03-26 09:34:29+01:00 Scheduling next slot start action block    tid=4794 fromNow=3 slot=39 node=4
DBG 2019-03-26 09:34:29+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4794 remote=Node[127.0.0.1:50006] node=4
INF 2019-03-26 09:34:29+01:00 GossipSub Peer disconnected                tid=4794 peer=Node[127.0.0.1:50000] reason=BreachOfProtocol node=4
Traceback (most recent call last)
beacon_node.nim(785)     beacon_node
beacon_node.nim(704)     start
beacon_node.nim(676)     run
asyncloop.nim(808)       runForever
asyncloop.nim(260)       poll
asyncmacro2.nim(36)      handshakeImpl_continue
rlpx.nim(1115)           handshakeImplIter
asyncmacro2.nim(306)     disconnectAndRaise
asyncmacro2.nim(36)      disconnectAndRaise_continue
rlpx.nim(51)             disconnectAndRaiseIter
asyncmacro2.nim(306)     disconnect
asyncmacro2.nim(36)      disconnect_continue
rlpx.nim(1139)           disconnectIter
rlpx.nim(1102)           callDisconnectHandlers
asyncmacro2.nim(306)     GossipSubPeerDisconnect
asyncmacro2.nim(36)      GossipSubPeerDisconnect_continue
gossipsub_protocol.nim(41) GossipSubPeerDisconnectIter
DBG 2019-03-26 09:34:29+01:00 Exception in rlpxConnect                   topics="rlpx" tid=4794 err="Future still in progress.\nAsync traceback:\n  beacon_node.nim(785)   beacon_node\n  beacon_node.nim(704)   start\n  beacon_node.nim(676)   run\n  asyncloop.nim(808)     runForever\n  asyncloop.nim(260)     poll\n  asyncmacro2.nim(36)    handshakeImpl_continue\n  rlpx.nim(1119)         handshakeImplIter\n  asyncfutures2.nim(329) read\n  #[\n    beacon_node.nim(785)   beacon_node\n    beacon_node.nim(704)   start\n    beacon_node.nim(676)   run\n    asyncloop.nim(808)     runForever\n    asyncloop.nim(260)     poll\n    asyncmacro2.nim(36)    BeaconSyncHandshake_continue\n    sync_protocol.nim(87)  BeaconSyncHandshakeIter\n    asyncfutures2.nim(324) read\n  ]#\n  #[\n    beacon_node.nim(785)   beacon_node\n    beacon_node.nim(704)   start\n    beacon_node.nim(676)   run\n    asyncloop.nim(808)     runForever\n    asyncloop.nim(260)     poll\n    asyncmacro2.nim(36)    postHelloSteps_continue\n    rlpx.nim(1208)         postHelloStepsIter\n    asyncfutures2.nim(324) read\n  ]#\nException message: Future still in progress.\nException type:" exc=ValueError remote=Node[127.0.0.1:50000] node=4
DBG 2019-03-26 09:34:29+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4613 remote=Node[127.0.0.1:50004] node=0
ERR 2019-03-26 09:34:29+01:00 dispatchMessages failed                    topics="rlpx" tid=4794 err="Data incomplete!\nAsync traceback:\n  beacon_node.nim(785)   beacon_node\n  beacon_node.nim(704)   start\n  beacon_node.nim(676)   run\n  asyncloop.nim(808)     runForever\n  asyncloop.nim(260)     poll\n  asyncmacro2.nim(36)    readExactly_continue\n  stream.nim(1284)       readExactlyIter\n  #[\n    beacon_node.nim(785)   beacon_node\n    beacon_node.nim(704)   start\n    beacon_node.nim(676)   run\n    asyncloop.nim(808)     runForever\n    asyncloop.nim(260)     poll\n    asyncmacro2.nim(36)    recvMsg_continue\n    rlpx.nim(404)          recvMsgIter\n    asyncfutures2.nim(324) read\n  ]#\nException message: Data incomplete!\nException type:" node=4
ERR 2019-03-26 09:34:29+01:00 dispatchMessages failed                    topics="rlpx" tid=4613 err="Data incomplete!\nAsync traceback:\n  beacon_node.nim(785)   beacon_node\n  beacon_node.nim(704)   start\n  beacon_node.nim(676)   run\n  asyncloop.nim(808)     runForever\n  asyncloop.nim(260)     poll\n  asyncmacro2.nim(36)    readExactly_continue\n  stream.nim(1284)       readExactlyIter\n  #[\n    beacon_node.nim(785)   beacon_node\n    beacon_node.nim(704)   start\n    beacon_node.nim(676)   run\n    asyncloop.nim(808)     runForever\n    asyncloop.nim(260)     poll\n    asyncmacro2.nim(36)    recvMsg_continue\n    rlpx.nim(404)          recvMsgIter\n    asyncfutures2.nim(324) read\n  ]#\nException message: Data incomplete!\nException type:" node=0
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice                  tid=4794 connectedPeers=1 node=4
INF 2019-03-26 09:34:33+01:00 Scheduling next slot start action block    tid=4779 fromNow=5 slot=40 node=8
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice                  tid=4788 connectedPeers=1 node=2
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice                  tid=4779 connectedPeers=1 node=8

hash_tree_root vs hash_tree_root_final

our code confuses the naming here vs spec with regards to zero padding - needs updating (hash_tree_root is the padded version in spec)

Resync with latest spec

Changelog: https://notes.ethereum.org/TQCF7Q07RI2tlP8Vmiv3RQ?view

v2.1 edits

add args validators and dynasty to get_active_validator_indices
add arg dynasty to get_new_shuffling
Added comment to ActiveState.recent_block_hashes dictating order
store 2*EPOCH_LENGTH hashes in ActiveState.recent_block_hashes
fixed parent_hashes calculation
Remove current_checkpoint
Added how/when to update crosslinks
"If justified_streak >= 65" --> "If justified_streak >= EPOCH_LENGTH + 1"
get_new_shuffling takes seed as an arg and passes it to shuffle
added Attestation validity condition -- slot > block.slot_number - EPOCH_LENGTH
Rename ShardAndIndices to ShardAndCommittee
Change var name ShardAndCommittee.validators to ShardAndCommittee.committee
A number of typos and remnants from previous spec

Implement bitfield data_structure for attester_bitfield

Currently we are using an IntSet for the attester_bitfield field. However this is not ideal.

IntSets

Cons

The SSZ serialization format of the bitfield is the raw packed bytes
IntSet uses an array of 34 ints on the stack and the rest on the heap. The first 34 ints are not bit-packed, the tail on the heap is packed. So we're always in the worst case scenario as the minimum recommended committee size is 111 and the TARGET_COMMITTEE_SIZE is 256

Nim builtin sets

Issues with Nim builtin sets

Taking the length of it requires iterating on all bits
It can hold at most 2^16 ints, the maximum committee size according to specs must be checked
Serialization is probably a binary dump but that must be tested

Pros

It's battle-tested and fast.
The length can be solved by using a custom object that tracks the length

type MyBitfield = object
  data: set[int16]
  len: int16

Do-It-Yourself bitfield

We can be sure that it works for our purposes but extra implementation, testing and maintenance time.

Python implementation is a short 36 lines here. Tests.

Note that the beacon chain specs mentions len(attester_bitfield) == ceil_div8(len(attestation_indices)) with ceil_div8 = (x + 7) // 8, This checks that the byte length of the bitfield is equal to the number of attesters divided by 8 and rounded up to the nearest multiple of 8. An alternative and arguably clearer implementation can be derived from Nimbus ceil32

[Windows 32-bit Perf] Passes through epoch update, empty block is slow

This part of the test suite is very slow and takes several minutes on Windows with beacon-chain built for 32-bit:

[OK] Passes through epoch update, empty block

https://github.com/status-im/nim-beacon-chain/blob/4747477160c60e456a43b837564934caa16198b5/tests/test_state_transition.nim#L66-L83

[RFC] epoch and slots - Distinct types

We have many issues related to epochs and slots and their unique treatment in the specs:

We probably need 4 distinct types. 2 for each epoch and slots.

type
  Slot = distinct uint64
  Epoch = distinct uint64
  PrintableSlot = distinct string
  PrintableEpoch = distinct string

With Slot and Epoch having no conversion to string defined and PrintableSlot/Epoch having only conversion to string defined.

[Crypto] Fix BLS domain

As evidenced by 43554ad in PR #80, many places are still using domain 0 in hash_to_G2 / sig verification.

This needs to be fixed. It is also very probably that several parts of Eth2 specs does not use domain.

cc @cheatfate

RFC: Logging levels

so regarding logging, to tune it so that operators can run INF and developers DBG, I would suggest the following guidelines:

NOT noteworthy events that are unusual as far as normal operation goes: (deep?) chain reorgs, slashings etc
INF logs anything that is sufficient to recreate the state along important checkpoints in the application, but does so in a minimal way, so as not to overwhelm log collectors - minimal in this sense means that it strives to avoid duplicating the same information - kind of like a normalized database - people running the app on a server should not be discouraged from always keeping INF on
DBG denormalizes the above and adds more checkpoints - repetition for convenience is ok, but we still keep it out of tight loops - developers will have this on by default
TRC anything goes

we can stick these in a document somewhere once we have a bit more agreement on the practical application of them, in nimbus specifically

Replace blake2s with blake2b-512[0 ..< 32]

Upstream spec change see ethereum/beacon_chain#60

[Simulation] Slot skipping

From time to time the simulation skips slots.

Example in https://gist.github.com/mratsim/1060f973b38c915db3172c27cc7e8e5f

Slot 6 and 8 were skipped. This can also happen later, in the same run slot 25 and 27 were skipped.

not all shards are being crosslinked on low validator counts, why?

same data as #150 - latest_crosslinks are being updated only for the first few shards - investigate why this is. notably, there's only 64 validators in that run which might be related

first validator has suspicious balance after running state sim

in state sim, with --attesterRatio=1 --slots=4096 everyone should be attesting - however, the first validator is getting penalized - investigate this:

https://twitter.com/jcksie/status/1101580136208629760

state.tar.gz

[Simulation] Epoch overflow to -1 (18446744073709551615)

DBG 2019-03-26 09:34:21+01:00 Block already exists                       tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 16, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=84786b22 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 16, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=8af56524 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=ac1f99bc node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
INF 2019-03-26 09:34:21+01:00 Scheduling next slot start action block    tid=4775 fromNow=18446744073709533 slot=34 node=6
INF 2019-03-26 09:34:21+01:00 Attestation sent                           tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b68d2663 validator=86d386aa node=6
INF 2019-03-26 09:34:21+01:00 Scheduling next slot start action block    tid=4775 fromNow=18446744073709539 slot=35 node=6
INF 2019-03-26 09:34:21+01:00 GossipSub Peer connected                   tid=4794 peer=Node[127.0.0.1:50006] node=4
INF 2019-03-26 09:34:21+01:00 Block proposed                             tid=4775 blck="(slot: 18, previous_block_root: \"723CF95D\", state_root: \"4F2093C3\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b13d1c9a\")" blockRoot=4300C585 idx=62 validator=86d386aa node=6
DBG 2019-03-26 09:34:21+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4792 remote=Node[127.0.0.1:50006] node=7
DBG 2019-03-26 09:34:21+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4794 remote=Node[127.0.0.1:50006] node=4
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 20, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b966bba9 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 20, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4794 attestationData="(slot: 16, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=92484f3c node=4
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=85894d4a node=6
DBG 2019-03-26 09:34:21+01:00 Growing attestation pool                   tid=4775 attestationSlot=22 startingSlot=0 node=6
ERR 2019-03-26 09:34:21+01:00 dispatchMessages failed                    topics="rlpx" tid=4794 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n  beacon_node.nim(785)       beacon_node\n  beacon_node.nim(704)       start\n  beacon_node.nim(676)       run\n  asyncloop.nim(808)         runForever\n  asyncloop.nim(260)         poll\n  asyncmacro2.nim(36)        dispatchMessages_continue\n  rlpx.nim(518)              dispatchMessagesIter\n  rlpx.nim(265)              invokeThunk\n  asyncmacro2.nim(306)       emit_thunk\n  asyncmacro2.nim(36)        emit_thunk_continue\n  rlpx.nim(769)              emit_thunkIter\n  asyncmacro2.nim(306)       emit\n  asyncmacro2.nim(36)        emit_continue\n  gossipsub_protocol.nim(61) emitIter\n  serialization.nim(65)      :anonymous\n  beacon_node.nim(666)       :anonymous\n  beacon_node.nim(638)       onAttestation\n  attestation_pool.nim(173)  add\n  attestation_pool.nim(71)   validate\n  validator.nim(150)         get_crosslink_committees_at_slot\n  system.nim(3879)           failedAssertImpl\n  system.nim(3872)           raiseAssert\n  system.nim(2918)           sysFatal\n  #[\n    beacon_node.nim(785)       beacon_node\n    beacon_node.nim(704)       start\n    beacon_node.nim(676)       run\n    asyncloop.nim(808)         runForever\n    asyncloop.nim(260)         poll\n    asyncmacro2.nim(36)        dispatchMessages_continue\n    rlpx.nim(518)              dispatchMessagesIter\n    rlpx.nim(265)              invokeThunk\n    asyncmacro2.nim(306)       emit_thunk\n    asyncmacro2.nim(39)        emit_thunk_continue\n    rlpx.nim(769)              emit_thunkIter\n    asyncfutures2.nim(324)     read\n  ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch:INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
 2, Next epoch: 1\nException type:" node=4
INF 2019-03-26 09:34:21+01:00 GossipSub Peer disconnected                tid=4794 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=4

DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=9796577f node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=a3066e4b node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4775 attestationData="(slot: 19, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b950bc65 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved                       tid=4775 attestationData="(slot: 19, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=3 node=6
INF 2019-03-26 09:34:21+01:00 Attestation sent                           tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=99567d7e validator=91887afb node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received                       tid=4794 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=9796577f node=4
DBG 2019-03-26 09:34:21+01:00 Block received                             tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
ERR 2019-03-26 09:34:21+01:00 dispatchMessages failed                    topics="rlpx" tid=4794 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n  beacon_node.nim(785)       beacon_node\n  beacon_node.nim(704)       start\n  beacon_node.nim(676)       run\n  asyncloop.nim(808)         runForever\n  asyncloop.nim(260)         poll\n  asyncmacro2.nim(36)        dispatchMessages_continue\n  rlpx.nim(518)              dispatchMessagesIter\n  rlpx.nim(265)              invokeThunk\n  asyncmacro2.nim(306)       emit_thunk\n  asyncmacro2.nim(36)        emit_thunk_continue\n  rlpx.nim(769)              emit_thunkIter\n  asyncmacro2.nim(306)       emit\n  asyncmacro2.nim(36)        emit_continue\n  gossipsub_protocol.nim(61) emitIter\n  serialization.nim(65)      :anonymous\n  beacon_node.nim(666)       :anonymous\n  beacon_node.nim(638)       onAttestation\n  attestation_pool.nim(173)  add\n  attestation_pool.nim(71)   validate\n  validator.nim(150)         get_crosslink_committees_at_slot\n  system.nim(3879)           failedAssertImpl\n  system.nim(3872)           raiseAssert\n  system.nim(2918)           sysFatal\n  #[\n    beacon_node.nim(785)       beacon_node\n    beacon_node.nim(704)       start\n    beacon_node.nim(676)       run\n    asyncloop.nim(808)         runForever\n    asyncloop.nim(260)         poll\n    asyncmacro2.nim(36)        dispatchMessages_continue\n    rlpx.nim(518)              dispatchMessagesIter\n    rlpx.nim(265)              invokeThunk\n    asyncmacro2.nim(306)       emit_thunk\n    asyncmacro2.nim(39)        emit_thunk_continue\n    rlpx.nim(769)              emit_thunkIter\n    asyncfutures2.nim(324)     read\n  ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=4
INF 2019-03-26 09:34:21+01:00 GossipSub Peer disconnected                tid=4794 peer=Node[127.0.0.1:50006] reason=ClientQuitting node=4

INF 2019-03-26 09:34:23+01:00 Preparing for fork choice                  tid=4775 connectedPeers=3 node=6
DBG 2019-03-26 09:34:23+01:00 Replaying state transitions                tid=4775 ancestors=1 prevStateSlot=0 stateRoot=246EF52C stateSlot=12 node=6
DBG 2019-03-26 09:34:24+01:00 Advancing state with empty slots           tid=4794 stateSlot=6 targetSlot=10 node=4
INF 2019-03-26 09:34:24+01:00 Storing state                              tid=4775 stateRoot=246EF52C stateSlot=0 node=6
DBG 2019-03-26 09:34:24+01:00 No head update this time                   tid=4775 headBlockRoot=723CF95D headBlockSlot=13 node=6
NOT 2019-03-26 09:34:24+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=4775 expectedEpoch=4 expectedSlot=37 fromNow=3 headSlot=13 node=6
INF 2019-03-26 09:34:24+01:00 Scheduling next slot start action block    tid=4775 fromNow=3 slot=38 node=6
DBG 2019-03-26 09:34:24+01:00 Attestation received                       tid=4775 attestationData="(slot: 23, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=8033bc50 node=6
ERR 2019-03-26 09:34:24+01:00 dispatchMessages failed                    topics="rlpx" tid=4775 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n  beacon_node.nim(785)       beacon_node\n  beacon_node.nim(704)       start\n  beacon_node.nim(676)       run\n  asyncloop.nim(808)         runForever\n  asyncloop.nim(260)         poll\n  asyncmacro2.nim(39)        dispatchMessages_continue\n  rlpx.nim(518)              dispatchMessagesIter\n  rlpx.nim(265)              invokeThunk\n  asyncmacro2.nim(306)       emit_thunk\n  asyncmacro2.nim(36)        emit_thunk_continue\n  rlpx.nim(769)              emit_thunkIter\n  asyncmacro2.nim(306)       emit\n  asyncmacro2.nim(36)        emit_continue\n  gossipsub_protocol.nim(61) emitIter\n  serialization.nim(65)      :anonymous\n  beacon_node.nim(666)       :anonymous\n  beacon_node.nim(638)       onAttestation\n  attestation_pool.nim(173)  add\n  attestation_pool.nim(71)   validate\n  validator.nim(150)         get_crosslink_committees_at_slot\n  system.nim(3879)           failedAssertImpl\n  system.nim(3872)           raiseAssert\n  system.nim(2918)           sysFatal\n  #[\n    beacon_node.nim(785)       beacon_node\n    beacon_node.nim(704)       start\n    beacon_node.nim(676)       run\n    asyncloop.nim(808)         runForever\n    asyncloop.nim(260)         poll\n    asyncmacro2.nim(39)        dispatchMessages_continue\n    rlpx.nim(518)              dispatchMessagesIter\n    rlpx.nim(265)              invokeThunk\n    asyncmacro2.nim(306)       emit_thunk\n    asyncmacro2.nim(39)        emit_thunk_continue\n    rlpx.nim(769)              emit_thunkIter\n    asyncfutures2.nim(324)     read\n  ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=6
INF 2019-03-26 09:34:24+01:00 GossipSub Peer disconnected                tid=4775 peer=Node[127.0.0.1:50008] reason=ClientQuitting node=6

DBG 2019-03-26 09:34:28+01:00 Block already exists                       tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block received                             tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block already exists                       tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block received                             tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block already exists                       tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Attestation received                       tid=4775 attestationData="(slot: 16, beacon_block_root: \"DB12D744\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=abc75cff node=6
ERR 2019-03-26 09:34:28+01:00 dispatchMessages failed                    topics="rlpx" tid=4775 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n  beacon_node.nim(785)       beacon_node\n  beacon_node.nim(704)       start\n  beacon_node.nim(676)       run\n  asyncloop.nim(808)         runForever\n  asyncloop.nim(260)         poll\n  asyncmacro2.nim(39)        dispatchMessages_continue\n  rlpx.nim(518)              dispatchMessagesIter\n  rlpx.nim(265)              invokeThunk\n  asyncmacro2.nim(306)       emit_thunk\n  asyncmacro2.nim(36)        emit_thunk_continue\n  rlpx.nim(769)              emit_thunkIter\n  asyncmacro2.nim(306)       emit\n  asyncmacro2.nim(36)        emit_continue\n  gossipsub_protocol.nim(61) emitIter\n  serialization.nim(65)      :anonymous\n  beacon_node.nim(666)       :anonymous\n  beacon_node.nim(638)       onAttestation\n  attestation_pool.nim(173)  add\n  attestation_pool.nim(71)   validate\n  validator.nim(150)         get_crosslink_committees_at_slot\n  system.nim(3879)           failedAssertImpl\n  system.nim(3872)           raiseAssert\n  system.nim(2918)           sysFatal\n  #[\n    beacon_node.nim(785)       beacon_node\n    beacon_node.nim(704)       start\n    beacon_node.nim(676)       run\n    asyncloop.nim(808)         runForever\n    asyncloop.nim(260)         poll\n    asyncmacro2.nim(39)        dispatchMessages_continue\n    rlpx.nim(518)              dispatchMessagesIter\n    rlpx.nim(265)              invokeThunk\n    asyncmacro2.nim(306)       emit_thunk\n    asyncmacro2.nim(39)        emit_thunk_continue\n    rlpx.nim(769)              emit_thunkIter\n    asyncfutures2.nim(324)     read\n  ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=6
INF 2019-03-26 09:34:28+01:00 GossipSub Peer disconnected                tid=4775 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=6

Block proposer node can crash when receiving its own proposal

After fixing #101 via #123 and other epochs issues (#105, #118).

Here is the latest one, a node can:

propose a block
receive its own proposal a couple of slots later
crash with unexpected block slot.

With the assert being at the soon infamous :) "Could this fail somehow"

https://github.com/status-im/nim-beacon-chain/blob/e1efdd4349a35903861a5384df0a96f07005fa56/beacon_chain/beacon_node.nim#L220-L229

log configuration/data directory on startup

print what folder is being used (~/.cache/nimbus/...) exactly, so user knows where to look

State slot can get stuck

The simulation can (always ?) get stuck in a state slot as of 2d52d5c

Debian Linux 32bit simulation failure

Wrote /home/cheatfate/nim-beacon-chain/tests/simulation/data/startup.json
Wrote /home/cheatfate/nim-beacon-chain/tests/simulation/data/state_snapshot.json
INF 2019-02-14 06:45:49-06:00 Initializing DB                            tid=23528
beacon_node.nim(427)     beacon_node
beacon_node.nim(61)      init
beacon_chain_db.nim(61)  persistState
beacon_chain_db.nim(45)  lastFinalizedState
Error: unhandled exception: Internal error: Database has no finalized state [Exception]

[Simulation] Epoch issue

An additional issue on top of #79 that impacts everyone and not only windows:

DBG 2019-02-07 12:07:34+01:00 Attaching validator                        thread=0 idx=0 pubKey=02c6d059784c69da51cc5f6c878a0852722186fce95f6644e86631e7f8ab332b7a05c1f8168d23a8399b107d4f33db2e validator=02c6d059 node=0
DBG 2019-02-07 12:07:34+01:00 Attaching validator                        thread=0 idx=1 pubKey=0b54490fe9f66437ce21e2f6bfb4324e4970b347d5543de47250c72fcf7f2f74e077efd5d45ed3b94b31a5c9d3953d8f validator=0b54490f node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=2 pubKey=0ee02e4857640286c946dac512e09b9e2f045431262f942f90aa6d7a12f8ddb4629b74153d8cc64fb965bbb5231888b9 validator=0ee02e48 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=3 pubKey=13921516d60bd2fd107e888afcfb363191d9d1ab1594f8ef57db8d016c7884f5252b8df2be01f3c11d3f4e7b89a4e37d validator=13921516 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=4 pubKey=8a538ee663cb9691e7f0dc82d88e3c0dbf4ef92c77581e93f98a81879a865a2db006523ebea52752fe06b80b145a63b1 validator=8a538ee6 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=5 pubKey=8473b922c54af034d3875ace14b3abc0c2ff994e1b05ef2ce84da908ae2c4c5f1a6d3cf0a62ba22a483361c9035e8bb9 validator=8473b922 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=6 pubKey=13830d6dd70c9daa9dcfeecc3bf920a914462c0ad0e34c176bd14b30567e2c7738cd25306f9611a3b5bdee69e8adafa0 validator=13830d6d node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=7 pubKey=85ea693dc32596b6763f6b7f6cbcde0d5b34c84cc8d4e54c3bd2cf2520a42ff4c37086bcfd32077ca12bffc1bb8d5c6f validator=85ea693d node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator                        thread=0 idx=8 pubKey=88e100a98e4f3b28f8128a02debab5b7e4cc7716642d797f8f36679fe5be170af9965e4201fc1a735bfe2af4ccc7c793 validator=88e100a9 node=0
INF 2019-02-07 12:07:35+01:00 Local validators attached                  thread=0 count=9 node=0
DBG 2019-02-07 12:07:35+01:00 Scheduling epoch actions                   thread=0 epoch=2 node=0
Current epoch: 2
Next epoch: 2
Current epoch: 144115188075855872 # 2^57 ~ Genesis block starts at 2^63 / 64 = 2^57
Next epoch: 1
beacon_node.nim(431)     beacon_node
beacon_node.nim(393)     processBlocks
beacon_node.nim(302)     scheduleEpochActions
validator.nim(89)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /Users/tesuji/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(89, 10) `previous_epoch <= epoch`  [AssertionError]

This was introduced following the latest spec changes, probably the introduction of get_crosslink_committees_at_slot and the removal of get_shard_committees_at_slot.

2f96c4b

Add Block Processing Benchmark

See alternative implementation in Lighthouse (Rust):

https://github.com/sigp/lighthouse/tree/bench_block_validation

And related issue in Prysm (Go):

prysmaticlabs/prysm#573

IndexError in shuffling when running sim with 1000 validators

859aa32

[arnetheduck@tempus research]$ nim c --opt:speed state_sim ; ./state_sim --attesterRatio=1 --slots=128 --validators=1000
...

confutils.nim(362)       state_sim
state_sim.nim(74)        CLI
testutil.nim(82)         addBlock
validator.nim(273)       get_beacon_proposer_index
validator.nim(243)       get_crosslink_committees_at_slot
validator.nim(132)       get_shuffling
system.nim(2830)         sysFatal
Error: unhandled exception: index out of bounds [IndexError]

Address comments from #46

cc @tersec @arnetheduck

socket error while sending crashes beacon node

13c7f7f

beacon_node.nim(779)     beacon_node
beacon_node.nim(673)     run
asyncloop.nim(756)	 runForever
asyncloop.nim(238)	 poll
asyncmacro2.nim(36)	 sendMsg_continue
rlpx.nim(285)            sendMsgIter
asyncfutures2.nim(324)   read
[[reraised from:
beacon_node.nim(779)     beacon_node
beacon_node.nim(673)     run
asyncloop.nim(756)	 runForever
asyncloop.nim(238)	 poll
asyncmacro2.nim(39)	 sendMsg_continue
excpt.nim(145)           sendMsgIter
]]
[[reraised from:
beacon_node.nim(779)     beacon_node
beacon_node.nim(673)     run
asyncloop.nim(756)	 runForever
asyncloop.nim(238)	 poll
asyncfutures2.nim(369)   cb
]]
Error: unhandled exception: (32) Broken pipe
Async traceback:
  beacon_node.nim(779)   beacon_node
  beacon_node.nim(673)   run
  asyncloop.nim(756)     runForever
  asyncloop.nim(238)     poll
  asyncmacro2.nim(36)    sendMsg_continue
  rlpx.nim(285)          sendMsgIter
  asyncfutures2.nim(324) read
  #[
    beacon_node.nim(779)   beacon_node
    beacon_node.nim(673)   run
    asyncloop.nim(756)     runForever
    asyncloop.nim(238)     poll
    asyncmacro2.nim(39)    sendMsg_continue
    excpt.nim(145)         sendMsgIter
  ]#
Exception message: (32) Broken pipe
Exception type: [Exception]

Unexpected crosslink shard failure in network sim

As of f9e0418, the local sim is relatively stable, but node fail occasionally with the following error:

WRN 2019-03-28 10:52:13+02:00 Unexpected crosslink shard                 tid=22764 acceptable_crosslink_data="@[(epoch: 536870914, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000), (epoch: 536870916, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)]" actual_epoch=4 attestation_data_previous_crosslink="(epoch: 536870914, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)" crosslink_data_root=0000000000000000000000000000000000000000000000000000000000000000 epoch=4 state_latest_crosslinks_attestation_data_shard="(epoch: 536870915, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)" node=8
beacon_node.nim(809)     beacon_node
beacon_node.nim(734)     start
beacon_node.nim(707)     run
asyncloop.nim(808)       runForever
asyncloop.nim(260)       poll
beacon_node.nim(658)     :anonymous
asyncmacro2.nim(306)     onSlotStart
asyncmacro2.nim(36)      onSlotStart_continue
beacon_node.nim(618)     onSlotStartIter
asyncmacro2.nim(306)     handleProposal
asyncmacro2.nim(36)      handleProposal_continue
beacon_node.nim(511)     handleProposalIter
asyncmacro2.nim(306)     proposeBlock
asyncmacro2.nim(36)      proposeBlock_continue
beacon_node.nim(359)     proposeBlockIter
system.nim(3790)         failedAssertImpl
system.nim(3783)         raiseAssert
system.nim(2830)         sysFatal
[[reraised from:
beacon_node.nim(809)     beacon_node
beacon_node.nim(734)     start
beacon_node.nim(707)     run
asyncloop.nim(808)       runForever
asyncloop.nim(260)       poll
beacon_node.nim(658)     :anonymous
asyncmacro2.nim(306)     onSlotStart
asyncmacro2.nim(36)      onSlotStart_continue
beacon_node.nim(618)     onSlotStartIter
asyncmacro2.nim(306)     handleProposal
asyncmacro2.nim(39)      handleProposal_continue
beacon_node.nim(511)     handleProposalIter
asyncfutures2.nim(324)   read
]]
[[reraised from:
beacon_node.nim(809)     beacon_node
beacon_node.nim(734)     start
beacon_node.nim(707)     run
asyncloop.nim(808)       runForever
asyncloop.nim(260)       poll
beacon_node.nim(658)     :anonymous
asyncmacro2.nim(306)     onSlotStart
asyncmacro2.nim(39)      onSlotStart_continue
beacon_node.nim(618)     onSlotStartIter
asyncfutures2.nim(324)   read
]]
[[reraised from:
beacon_node.nim(809)     beacon_node
beacon_node.nim(734)     start
beacon_node.nim(707)     run
asyncloop.nim(808)       runForever
asyncloop.nim(260)       poll
asyncfutures2.nim(369)   cb
]]
Error: unhandled exception: /home/zahary/nimbus/vendor/nim-beacon-chain/beacon_chain/beacon_node.nim(359, 12) `ok` 
Async traceback:
  beacon_node.nim(809)   beacon_node
  beacon_node.nim(734)   start
  beacon_node.nim(707)   run
  asyncloop.nim(808)     runForever
  asyncloop.nim(260)     poll
  beacon_node.nim(658)   :anonymous
  asyncmacro2.nim(306)   onSlotStart
  asyncmacro2.nim(36)    onSlotStart_continue
  beacon_node.nim(618)   onSlotStartIter
  asyncmacro2.nim(306)   handleProposal
  asyncmacro2.nim(36)    handleProposal_continue
  beacon_node.nim(511)   handleProposalIter
  asyncmacro2.nim(306)   proposeBlock
  asyncmacro2.nim(36)    proposeBlock_continue
  beacon_node.nim(359)   proposeBlockIter
  system.nim(3790)       failedAssertImpl
  system.nim(3783)       raiseAssert
  system.nim(2830)       sysFatal
  #[
    beacon_node.nim(809)   beacon_node
    beacon_node.nim(734)   start
    beacon_node.nim(707)   run
    asyncloop.nim(808)     runForever
    asyncloop.nim(260)     poll
    beacon_node.nim(658)   :anonymous
    asyncmacro2.nim(306)   onSlotStart
    asyncmacro2.nim(36)    onSlotStart_continue
    beacon_node.nim(618)   onSlotStartIter
    asyncmacro2.nim(306)   handleProposal
    asyncmacro2.nim(39)    handleProposal_continue
    beacon_node.nim(511)   handleProposalIter
    asyncfutures2.nim(324) read
  ]#
  #[
    beacon_node.nim(809)   beacon_node
    beacon_node.nim(734)   start
    beacon_node.nim(707)   run
    asyncloop.nim(808)     runForever
    asyncloop.nim(260)     poll
    beacon_node.nim(658)   :anonymous
    asyncmacro2.nim(306)   onSlotStart
    asyncmacro2.nim(39)    onSlotStart_continue
    beacon_node.nim(618)   onSlotStartIter
    asyncfutures2.nim(324) read
  ]#
Exception message: /home/zahary/nimbus/vendor/nim-beacon-chain/beacon_chain/beacon_node.nim(359, 12) `ok` 
Exception type: [AssertionError]

Error: could not load: librocksdb(|_lite).dylib

(On MacOS)
(using the code on -master)

$ nimble build
$ nimble test
both work/pass.

But when running:

$ ./tests/simulation/start.sh

I get the following error:

could not load: librocksdb(|_lite).dylib

Do I need to manually install librocksdb?

Shuffling tests

The cutoffs function description is the followinf:

An algorithm that we use to split up validators into groups at the start of every epoch, determining at what height they can make attestations and what shard they are making crosslinks for.

The schema:

A full Python implementation:

SHARD_COUNT = 1024
ETH_SUPPLY_CAP = 2**27
DEPOSIT_SIZE = 32
MAX_VALIDATOR_COUNT = ETH_SUPPLY_CAP // DEPOSIT_SIZE
EPOCH_LENGTH = 64
SLOT_DURATION = 8
MIN_COMMITTEE_SIZE = 128
END_EPOCH_GRACE_PERIOD = 8

def get_cutoffs(validator_count):
    height_cutoffs = [0]
    # EPOCH_LENGTH // phi
    cofactor = 39
    STANDARD_COMMITTEE_SIZE = MAX_VALIDATOR_COUNT // SHARD_COUNT
    # If there are not enough validators to fill a minimally
    # sized committee at every height, skip some heights
    if validator_count < EPOCH_LENGTH * MIN_COMMITTEE_SIZE:
        height_count = validator_count // MIN_COMMITTEE_SIZE or 1
        heights = [(i * cofactor) % EPOCH_LENGTH
                   for i in range(height_count)]
    # If there are enough validators, fill all the heights
    else:
        height_count = EPOCH_LENGTH
        heights = list(range(EPOCH_LENGTH))

    filled = 0
    for i in range(EPOCH_LENGTH - 1):
        if not i in heights:
            height_cutoffs.append(height_cutoffs[-1])
        else:
            filled += 1
            height_cutoffs.append(filled * validator_count // height_count)
    height_cutoffs.append(validator_count)

    # For the validators assigned to each height, split them up
    # into committees for different shards. Do not assign the
    # last END_EPOCH_GRACE_PERIOD heights in an epoch to any shards.
    shard_cutoffs = [0]
    for i in range(EPOCH_LENGTH - END_EPOCH_GRACE_PERIOD):
        size = height_cutoffs[i+1] - height_cutoffs[i]
        shards = (size + STANDARD_COMMITTEE_SIZE - 1) // STANDARD_COMMITTEE_SIZE
        pre = shard_cutoffs[-1]
        for j in range(1, shards+1):
            shard_cutoffs.append(pre + size * j // shards)

    return height_cutoffs, shard_cutoffs

(h, s) = get_cutoffs(16)

# Note: Python 2 because Apple/Mac ...
print height # [0, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16]
print len(height) # 65
print shard # [0, 16]

The Python implementation does not split like in the example
The example may be a simplification that shows the spirit of the spec but no sharding occurs because 16 validators is too small.

Nim implementation:
https://github.com/status-im/nim-beacon-chain/blob/585072ae154e3d10276b51397e707628a401c096/beacon_chain/private/helpers.nim#L35-L79

# to run from ./build
import ../beacon_chain/private/helpers
import ../beacon_chain/datatypes

let a = 16
var seed: Blake2_256_Digest

echo getShuffling(seed, a) # @[14, 9, 13, 10, 3, 6, 12, 1, 2, 0, 5, 15, 8, 7, 11, 4]
let (height, shard) = getCutoffs(a)

echo height # @[0, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16]
echo height.len # 65
echo shard # @[0, 16]

[HowTo] Allow simulation past 99 validators and 9 validator nodes

This was a bit tricky are many ranges are implicitly assumed in the code at the moment.

To allow simulation on more than 99 validators and 9 validators node you need to change 5 things:

tests/simulation/start.sh
- padding in the enumerator
- padding in the master node address
- padding in the tcp/udp port
- padding in the beacon node CLI call for validator config
validator_keygen.nim
- padding in validator keygen generated json

Example commits in https://github.com/status-im/nim-beacon-chain/tree/allow-more-than-99-validators

[Windows Install] Mention how to install rocksdb

[Simulation] findAncestorBySlot SIGSEGV in blockpool

This is on branch #192

The full log is

[Testnet] Fails to download key and load network file

INF 2019-03-27 02:40:06+01:00 New testnet genesis data received. Starting with a fresh database. tid=12761
Failed to download key
cannot open: /home/swader/.cache/nimbus/BeaconNode/testnet0/network.jsonswader@swader-N76VZ:~/repos/nimbus/vendor/nim-beacon-chain$

This when restarting testnet join without cache etc. rm -rf ~/.cache/nimbus et al.

[Windows] Simulation doesn't work

Explored in this branch https://github.com/status-im/nim-beacon-chain/tree/fix-sim-init-windows

The shards and committee were never populated.
Latest changes removed them, which might fix the issue.

Unfortunately this is pending a fix or workaround for #78

Assertion at slot 8192

INF 2019-03-08 01:58:04-06:00 Storing state                              tid=11707 stateRoot=34568B2765DF7853F87D9DEF88F27A83DDD421FE8C2F44D6EA78044DC704871C stateSlot=8192 node=9
INF 2019-03-08 01:58:04-06:00 Updated head                               tid=11707 headBlockRoot=44E30AF2 stateRoot=34568B27 stateSlot=8192 node=9
DBG 2019-03-08 01:58:04-06:00 Advancing state past slot gap              tid=11707 stateSlot=8192 targetSlot=8193 node=9
INF 2019-03-08 01:58:04-06:00 Scheduling next slot start action block    tid=11707 fromNow=5 slot=8194 node=9
DBG 2019-03-08 01:58:04-06:00 No new head found                          tid=11707 blockRoot=44E30AF2 stateRoot=34568B27 stateSlot=8192 node=9
DBG 2019-03-08 01:58:04-06:00 Advancing state past slot gap              tid=11707 stateSlot=8192 targetSlot=8193 node=9
beacon_node.nim(667)     beacon_node
beacon_node.nim(621)     run
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(417)     :anonymous
asyncmacro2.nim(306)     makeAttestation
asyncmacro2.nim(36)      makeAttestation_continue
beacon_node.nim(270)     makeAttestationIter
beaconstate.nim(278)     get_block_root
system.nim(3790)         failedAssertImpl
system.nim(3783)         raiseAssert
system.nim(2830)         sysFatal
[[reraised from:
beacon_node.nim(667)     beacon_node
beacon_node.nim(621)     run
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
asyncfutures2.nim(370)   cb
]]
Error: unhandled exception: /home/arnetheduck/status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim(278, 12) `state.slot <= slot + 8192` 
Async traceback:
  beacon_node.nim(667) beacon_node
  beacon_node.nim(621) run
  asyncloop.nim(756)   runForever
  asyncloop.nim(238)   poll
  beacon_node.nim(417) :anonymous
  asyncmacro2.nim(306) makeAttestation
  asyncmacro2.nim(36)  makeAttestation_continue
  beacon_node.nim(270) makeAttestationIter
  beaconstate.nim(278) get_block_root
  system.nim(3790)     failedAssertImpl
  system.nim(3783)     raiseAssert
  system.nim(2830)     sysFatal

I suspect it's related to us not finalizing stuff, and at the same time reaching the limit of some state history field

31a478e

Gossipsub: randomize peer id to avoid trivially linking validator to libp2p transport information

With regards to ethereum/consensus-specs#593

Gossipsub messages are signed by the peer id / peer public key (libp2p/specs#140), meaning it's possible to link an attestation to a peer id, and therefore trivially lookup transport details of a validator.

@raulk:

The peer ID is a hash of the public key. If you’re using a recoverable signature, you should be able to match on pubkey, but once you have the pubkey, the conversion to a peer ID is trivial

[Testing] Compile-time config change of the beacon chain

Discussing with @djrtwo, for testing purposes beacon chains will have a couple of different configurations of the "constants" in the spec.

For example SLOTS_PER_EPOCH is 8 here: https://github.com/ethereum/eth2.0-tests/blob/0d4bb97d71e3b7e983fa5c0cb22039e2aba26b1a/state/sanity-check_small-config_32-vals.yaml#L27

but 64 here: https://github.com/ethereum/eth2.0-tests/blob/0d4bb97d71e3b7e983fa5c0cb22039e2aba26b1a/state/sanity-check_default-config_100-vals.yaml

Solutions

1. Changing to runtime values

This would need rewriting lots of the code from array to sequences. Also it will prevents us many optimizations like preallocating a buffer or avoiding GC memory.

2. Have config1/config2/config3/... constant files

This is the easier solution has only a restricted amount of configs are needed.

Example:

const constantsConfig {.strdefine.} = "official"

when constantsConfig == "official":
  import constants/official
elif constantsConfig == "test1":
  import constants/test1

and compile the relevant tests with nim c -d:constantsConfig=test1 beacon_node

3. Read the config part of the YAML files at compile-time

This is the most flexible but quite involved (overkill?) as it would require a compile-time yaml parser and then a macro that generates the proper constants during compilation.

[Simulation] End of epoch issue

After the fixing of #85 and #89, we now have a new issue during epoch transition.

Reproduced also on @yglukhov side:

With added logs on epochs and slots:
- in scheduleEpochActions -- https://github.com/status-im/nim-beacon-chain/blob/2794181889801c17f6d9ecca180e04c9bada29f7/beacon_chain/beacon_node.nim#L275-L310
- in get_crosslink_committees_at_slot -- https://github.com/status-im/nim-beacon-chain/blob/2794181889801c17f6d9ecca180e04c9bada29f7/beacon_chain/spec/validator.nim#L69-L80

fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:24:56+01:00 Waiting for connections                    thread=0 node=0
INF 2019-02-07 19:24:56+01:00 RLPx listener up                           thread=0 self=enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000 node=0
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=3
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=2
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://18ceedebc1034b2d97fe1a3dd20f4cb1de7e740b0dd59fd80425cfeec2e2f07b07639637294e5d406f71f2fcbf91160fa09637ca30f4a4f063628bf484ac08a0@127.0.0.1:50002 node=2
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=2
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://14b6f6abf2de6256e82fc5961cdd545cef2d1800d5d3d5c6fd311d85deccfb2ddb45b1e4cfb2dda31edb08634fddd4535176c5c2024d0b14866b4ae7e6ba1f44@127.0.0.1:50003 node=3
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=3
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=1
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=6
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://252ba14c6ee47dfdf1c8931b8f47e2265ad257ec8d9430103ed7f9b6b14f687a1a816202256d147641c3059541a294b9efaeed2995646264ffc121eea5304325@127.0.0.1:50001 node=1
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=1
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=5
fork_slot: 9223372036854775808
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://56bde0fab36731a9b9920cd9ecbc66cf0b06f0a5096aac4dc16c60c8e3b4c15cf067ef92edaa91172f376acfce309368a41a9fdb3e89f458199112ceffa27536@127.0.0.1:50006 node=6
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=6
node slot: 0
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://dfe063548e8ac825d60cd0cd367273ad99c0c9547b800609ee5f6951ca0027167b8de026e16ab1b2d3c2964936df0649fff7d7e4502a34925cf44d5b58c1df3c@127.0.0.1:50005 node=5
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=5
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=9
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=7
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=4
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://bbe000c0a94aecd030402128b829bd12562e1bf4d8300c80a2769eb2b30eabc98a799d279cfce207a04360e139cc7858f32d7eaa53705320d15712c35834833c@127.0.0.1:50009 node=9
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=9
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes              thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=8
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://cdf6260a7ebdebd0e498088afebe8adf3201586db86ccf7668574ea21764fc33f1141153c869671808971ac3a44a13460bcbc5e9539c7a9f67ae7b5ce323bd78@127.0.0.1:50007 node=7
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=7
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://eb5cf6725dd129edb77f25405ae538183303ddce9155839a21c64fede1577ed4733f1538b2862ce97b6153220c07232c9d4181dada8174156bde151dbced6085@127.0.0.1:50008 node=8
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=8
INF 2019-02-07 19:25:02+01:00 RLPx listener up                           thread=0 self=enode://1a6c56c1cf9cb7bbfe3ddb18906876a7ee314958b745b1d0fefbd747013a94d90d9a0db1392b38f868627b9d97763acf67c6d7c96d2ea02d9569f579d5d7401e@127.0.0.1:50004 node=4
INF 2019-02-07 19:25:02+01:00 Bonding to peer                            topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=4
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=3
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=3
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=2
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=2
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=3
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=2
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=1
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=1
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=1
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=6
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=6
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=5
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=5
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=6
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=5
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=9
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=9
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=7
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=7
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=9
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=7
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=4
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from   topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=8
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=4
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes        topics="discovery" thread=0 node=8
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=8
DBG 2019-02-07 19:25:03+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=4
INF 2019-02-07 19:25:03+01:00 Starting beacon node                       thread=0 slotsSinceFinalization=-144115188075855872 stateSlot=9223372036854775808 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=0 pubKey=1547bfa50ff2dbcd3c6df014657310cbbd7262c9dcfd62875b085dfa947a20bf0bec359a64c2d41c06f8b61d1ad9ffdf validator=1547bfa5 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=1 pubKey=17f186031dede4d908aef7882445328bd98a390855dfce07fc12be73b90c77ddb0fdedd003ff23a1a101e6e842b07ef3 validator=17f18603 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=2 pubKey=8249e0535a6f3cfd206777e4057edbb83a927067c77955283663f105e44258fcdbf0e55c436551489df5ab1e20c9ef55 validator=8249e053 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=3 pubKey=0ca6764168121779715ee331e867f9da0c1d3b889b8d1a17abee3272b3c362933f3af2eba47dd8cac289a2f71bdeb4dd validator=0ca67641 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=4 pubKey=027fa75ec2ba5a3dc7b1ed314de771c7f08135e8d4c1cc6bb2d696669e7be4627fa570be348f38356596d5c5798bc874 validator=027fa75e node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=5 pubKey=988c71630e0722245655a2ba50269c3b95a4c2be3468bb577f2a21633c4bee361fbb00a03ce690086000ac1bf197a5bb validator=988c7163 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=6 pubKey=0a42564e2ac7bae576e62304a524f8a372b524f015543801c13d45fb8967f5da505a9cde19fbbeee87c3f49e88faf4a3 validator=0a42564e node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=7 pubKey=16efb9733a20a511fa4e1d4ed06a8594496a8424df7d5dae31e4ac14d87a88119a4382a26af8636375430b3525cd86db validator=16efb973 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator                        thread=0 idx=8 pubKey=010c62895e4194ec694033016128347c63946eb7b15a2cf5195c0260a02663435de8b8e9aeaf6f802ffc2bb11d073798 validator=010c6289 node=0
INF 2019-02-07 19:25:03+01:00 Local validators attached                  thread=0 count=9 node=0
DBG 2019-02-07 19:25:03+01:00 Genesis epoch                              thread=0 GENESIS_EPOCH=144115188075855872 node=0
DBG 2019-02-07 19:25:03+01:00 Scheduling epoch actions since genesis     thread=0 epoch=144115188075855872 node=0
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


i: 0
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


i: 1
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


i: 2
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


i: 3
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


[...]


i: 22
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


INF 2019-02-07 19:25:03+01:00 Scheduling block proposal                  thread=0 fromNow=4 slot=9223372036854775832 validator=027fa75e node=0
i: 23
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


[...]


i: 42
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


INF 2019-02-07 19:25:03+01:00 Scheduling block proposal                  thread=0 fromNow=124 slot=9223372036854775852 validator=1547bfa5 node=0
i: 43
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873

[...]

i: 49
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


INF 2019-02-07 19:25:03+01:00 Scheduling block proposal                  thread=0 fromNow=166 slot=9223372036854775859 validator=988c7163 node=0
i: 50
[...]


i: 56
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


INF 2019-02-07 19:25:03+01:00 Scheduling block proposal                  thread=0 fromNow=208 slot=9223372036854775866 validator=16efb973 node=0
i: 57
[...]


i: 61
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


i: 62
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


Input epoch: 144115188075855873
Previous epoch: 144115188075855872
Current epoch: 144115188075855873
Next epoch: 144115188075855874


i: 63
Input epoch: 144115188075855873
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873


beacon_node.nim(438)     beacon_node
beacon_node.nim(397)     processBlocks
beacon_node.nim(306)     scheduleEpochActions
validator.nim(86)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /Users/<PATH>/nim-beacon-chain/beacon_chain/spec/validator.nim(86, 10) `epoch < next_epoch`  [AssertionError]
DBG 2019-02-07 19:25:05+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=5
DBG 2019-02-07 19:25:05+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=9
DBG 2019-02-07 19:25:05+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=6
DBG 2019-02-07 19:25:05+01:00 Not enough nodes                           topics="discovery" thread=0 present=0 requested=1 node=1

Public key aggregation is slow

Sig aggregation has OK times but public key aggregation is very slow: status-im/nim-blscurve#13.

Pubkey aggregation might not be needed though. To be checked.

Unexpected crosslink shard in sim

21e4deb

Looks like an off-by-one somewhere, epoch numbers don't match:

WRN 2019-03-04 14:51:43-06:00 Unexpected crosslink shard                 tid=18654 attestation_data_latest_crosslink="(epoch: 1152921504606846992, shard_block_root: 0000000000000000000000000000000000000000000000000000000000000000)" epoch=18 shard_block_root=0000000000000000000000000000000000000000000000000000000000000000 state_latest_crosslinks_attestation_data_shard="(epoch: 1152921504606846993, shard_block_root: 0000000000000000000000000000000000000000000000000000000000000000)" node=5

node5.log

Testnet MVP

Networking

General idea is to switch beacon_node backend to libp2p using daemon and develop a simple protocol on top using SSZ as serialization.

Further, the idea is to introduce a management layer that contains logic for dealing with retrying requests and coordinating peer scoring. Basically, the attestation and block pools signal the hashes they need and a separate layer decides on the logic to fetch these from the peer layer (how many concurrent requests, when to retry the same block). When blocks arrive, either from broadcasts or requests, they should flow into the pool the same way.

Switch to libp2p via daemon (postponed)
Specify simple SSZ-based messages for network operations

State sync

When joining the testnet, client will be behind. We will regularly restart the testnet in the beginning, thus we primarily need to have the capability to catch up by "full sync" - downloading all blocks. The other case where blocks are needed is when an attestation or block is received, and the dependent blocks are not (lost in translation, missing history, unknown fork etc)

Block request (request by hash or equivalent)
State recovery (low prio)
State diff / light client (low prio)

Broadcast

After validating or proposing blocks, these will be (naively) gossiped all other participants so they can count votes and decide on forks. The most simple implementation idea seems to be to publish attestations with a single signature, then aggregate lazily as needed (for example when proposing a block).

Attestations
Proposer blocks

libp2p considerations

When switching to libp2p, make sure these issues are covered and used correctly, as a minimum:

Peer/service discovery, including features - use private libp2p network or piggyback on ipfs?
Version negotiation - spec version, protocol features

Fork management

Forks start with the latest finalized block and build a tree of possible futures from there. The idea is to manage known blocks and attestations as a collection, and take action to fill out that collection as needed.

There are many race conditions that all need to be handled gracefully, and the code should have room to modify the strategy for handling these:

attestation with unknown block
block with unknown parent
etc

One problem to consider is that of worst case performance, in case of malicious blocks being posted by validators (for example, lots of unviable forks / blocks causing data structure and network traffic growth)

Attestation pool
Block pool

Validator management

Adding and removing validators is somewhat in flux, so for now the plan is to not use the ETH1 contract for this feature. Initially, we'll just publish JSON files with validator data (priv key etc) and manage overlap socially. The majority will likely be used in pre-configured beacon nodes running on a server, while some will be reserved for developers to play with.

Potential issues include two people running the same validator - this is a feature as it will help us discover issues when this happens (for example if we receive an attestation signed by our own key that we did not send, this is a warning sign that the private key is being reused).

Share validator JSON, let people manage socially
Web service to add/remove validators (low prio)

Devops

We'll initially deploy one or more boot nodes on a server, each hosting a number of validators. The general idea is to restart the testnet frequently. People wanting to connect will get genesis and validator information from the server via.. whatever (http listing).

Servers & automated deployment
Monitoring (logs etc)
Extra points for having a Grafana or similar deployment, with graylog or elasticsearch/logstash collecting logs from the nodes, to be able to monitor the network real-time

Configurability

Don't wanna run a big network just yet 😄

Allow configuration of shard count etc, so as to create a smaller network

Spec updates

The general idea is to follow spec releases by updating every time there's a new upstream release.

Verify / review v0.5.1 compatibility to reach a stable point
Version / release nim-beacon-chain according to spec version it supports

Same state hash with different slots

Following the "Preparing for fork choice" logs, during the replay of state transition, the replay starts from the same state hash but seemingly at different slots.

Potential incorrect serialization of bls pub key

Issue

Note: I'm looking at Nim for the first time so this might just be incorrect.

The following defines a BLSPubKey which is used in the ValidatorRecord type definition
https://github.com/status-im/nim-beacon-chain/blob/ca4082b316689c8702dc8ba665611de31e3a92cf/beacon_chain/datatypes.nim#L63-L65

I believe that when you serialize BLSPubKey with SSZ that you will get a length prefix in the byte output. Because the length is known, we will likely use int384 or bytes48. Both of these types will not require a length prefix when serialized with SSZ.

EDIT
It looks like the items in this aggregateSig will suffer the same fate.
https://github.com/status-im/nim-beacon-chain/blob/ca4082b316689c8702dc8ba665611de31e3a92cf/beacon_chain/datatypes.nim#L77

Note, that while using the length of the BLSPubKey works for the bn256 curve for aggregate sig, if we use BLS12-381, the signatures are not the same length as the pub keys. One is 48bytes while the other is 96bytes (depending on if you G1 or G2 for the key).

BLS: update benchmarks to latest scheme

it would probably be good if benchmarks were expressed in terms of spec/crypto.nim..

Errors during beacon node simulation

These errors are being thrown by the straggler node that's run with run_node.sh 9 after the other 9 nodes are already running for 30 or so minutes. They happen rarely and are not easy to reliably reproduce - in fact, they do not happen again once the node is in sync with others.

This happens on Ubuntu 18.04.2.

INF 2019-03-12 15:52:12+01:00 Forward sync imported blocks               tid=22243 badBlocks=0 bodies=455 goodBlocks=455 headers=455 roots=455 node=9
DBG 2019-03-12 15:52:12+01:00 Exception in rlpxConnect                   topics="rlpx" tid=22243 err="Can\'t obtain a value from a `none`\nAsync traceback:\n  beacon_node.nim(627)   beacon_node\n  asyncloop.nim(761)     waitFor\n  asyncloop.nim(238)     poll\n  asyncmacro2.nim(36)    BeaconSyncHandshake_continue\n  sync_protocol.nim(108) BeaconSyncHandshakeIter\n  options.nim(132)       get\n  #[\n    beacon_node.nim(627)   beacon_node\n    asyncloop.nim(761)     waitFor\n    asyncloop.nim(238)     poll\n    asyncmacro2.nim(36)    postHelloSteps_continue\n    rlpx.nim(1206)         postHelloStepsIter\n    asyncfutures2.nim(325) read\n  ]#\nException message: Can\'t obtain a value from a `none`\nException type:" exc=UnpackError:ObjectType remote=Node[127.0.0.1:50001] node=9
INF 2019-03-12 15:52:12+01:00 Syncing state from remote peers            tid=22243 finalized_epoch=0 target_slot_epoch=72 node=9
INF 2019-03-12 15:52:12+01:00 Starting beacon node                       tid=22243 SECONDS_PER_SLOT=6 SHARD_COUNT=4 SLOTS_PER_EPOCH=8 SPEC_VERSION=0.4.0 slotsSinceFinalization=583 stateSlot=50 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=90 pubKey=14aa2583a999066ec6caa72a3fc19e80d8936f6856d447dd043aa9b126aa63bcaac876266d80913071777984d8d30563 validator=14aa2583 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=91 pubKey=0eb8b1b309a726fa5af6a6228385214a48788a1f23fe03cd46e16e200ed7d8909394d2e0b442ef71e519215765ca6625 validator=0eb8b1b3 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=92 pubKey=8c7b0e11f9bc3f48d84013ef8e8575aeb764bc1b9bf15938d19eb191201011365c2b14d78139a0f27327cb21c1b8bf3d validator=8c7b0e11 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=93 pubKey=0d08a52857017fd5cab3a821ccb8f5908c96cf63c5a5647209c037e2ea1c56f9650ec030b82ffdce76d37672d942e45b validator=0d08a528 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=94 pubKey=88f5540a9977fd2ee7dea836ed3dafa5d0b1fc9c5d5f1689e91ec49cdef989976c51502c3764025ef8ff542ef3b170ea validator=88f5540a node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=95 pubKey=8ff7cc69f007f11481c91c6f9b20698998a0c2e9a2928bec8eea7507c7ad73a9d1d218cfdb279c4d2132d7da6c9e513e validator=8ff7cc69 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=96 pubKey=0fb72b4c111da98379f195da4e5c18462acc7ece85cd66894fbaf69ddab3d3bb0b6957ea0042b7705937919189e6a531 validator=0fb72b4c node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=97 pubKey=812b2d0546aa77dec2d55406b0131ed580c079c1aeb76eb2ca076b7b58289fa9d781069a2e11fe2199f1e02c5dd70e6a validator=812b2d05 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=98 pubKey=8a10e1055b14a89cc3261699524998732fddc4f30c76c1057eb83732a01416643eb015a932e4080c86f42e485973d240 validator=8a10e105 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator                        tid=22243 idx=99 pubKey=029e520a73ec28f4e2e45050c93080eeaee57af1108e659d740897c3ced76ceb75d106cb00d7ed25ec221874bf4b235a validator=029e520a node=9
INF 2019-03-12 15:52:12+01:00 Local validators attached                  tid=22243 count=10 node=9
DBG 2019-03-12 15:52:12+01:00 Scheduling epoch actions                   tid=22243 epoch=72 stateEpoch=6 node=9
NOT 2019-03-12 15:52:13+01:00 Replaying state transitions                tid=22243 ancestors=3 prevStateSlot=552 stateRoot=02893E03 stateSlot=50 node=9
INF 2019-03-12 15:52:13+01:00 Updated head                               tid=22243 headBlockRoot=7A7528B2 stateRoot=22ABAD68 stateSlot=554 node=9
NOT 2019-03-12 15:52:13+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=22243 expectedEpoch=72 expectedSlot=583 fromNow=2 stateSlot=554 node=9
INF 2019-03-12 15:52:13+01:00 Scheduling next slot start action block    tid=22243 fromNow=2 slot=584 node=9
INF 2019-03-12 15:52:16+01:00 Scheduling next slot start action block    tid=22243 fromNow=5 slot=585 node=9
DBG 2019-03-12 15:52:16+01:00 Scheduling epoch actions                   tid=22243 epoch=73 stateEpoch=69 node=9
DBG 2019-03-12 15:52:16+01:00 No new head found                          tid=22243 blockRoot=7A7528B2 stateRoot=22ABAD68 stateSlot=554 node=9
NOT 2019-03-12 15:52:16+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=22243 expectedEpoch=73 expectedSlot=584 fromNow=5 stateSlot=554 node=9
DBG 2019-03-12 15:52:16+01:00 Attestation received                       tid=22243 beaconBlockRoot=7E0738DF justifiedBlockRoot=FA19A839 justifiedEpoch=70 shard=0 signature=0cb50812 slot=584 node=9
ERR 2019-03-12 15:52:16+01:00 dispatchMessages failed                    topics="rlpx" tid=22243 err="/home/swader/repos/nim-beacon-chain/beacon_chain/spec/validator.nim(153, 12) `epoch <= next_epoch` Previous epoch: 68, epoch: 73, Next epoch: 70\nAsync traceback:\n  beacon_node.nim(641)       beacon_node\n  beacon_node.nim(595)       run\n  asyncloop.nim(756)         runForever\n  asyncloop.nim(238)         poll\n  asyncmacro2.nim(36)        dispatchMessages_continue\n  rlpx.nim(519)              dispatchMessagesIter\n  rlpx.nim(266)              invokeThunk\n  asyncmacro2.nim(306)       emit_thunk\n  asyncmacro2.nim(36)        emit_thunk_continue\n  rlpx.nim(767)              emit_thunkIter\n  asyncmacro2.nim(306)       emit\n  asyncmacro2.nim(36)        emit_continue\n  gossipsub_protocol.nim(61) emitIter\n  serialization.nim(65)      :anonymous\n  beacon_node.nim(585)       :anonymous\n  beacon_node.nim(532)       onAttestation\n  attestation_pool.nim(246)  add\n  attestation_pool.nim(139)  validate\n  validator.nim(153)         get_crosslink_committees_at_slot\n  system.nim(3790)           failedAssertImpl\n  system.nim(3783)           raiseAssert\n  system.nim(2830)           sysFatal\n  #[\n    beacon_node.nim(641)       beacon_node\n    beacon_node.nim(595)       run\n    asyncloop.nim(756)         runForever\n    asyncloop.nim(238)         poll\n    asyncmacro2.nim(36)        dispatchMessages_continue\n    rlpx.nim(519)              dispatchMessagesIter\n    rlpx.nim(266)              invokeThunk\n    asyncmacro2.nim(306)       emit_thunk\n    asyncmacro2.nim(39)        emit_thunk_continue\n    rlpx.nim(767)              emit_thunkIter\n    asyncfutures2.nim(325)     read\n  ]#\nException message: /home/swader/repos/nim-beacon-chain/beacon_chain/spec/validator.nim(153, 12) `epoch <= next_epoch` Previous epoch: 68, epoch: 73, Next epoch: 70\nException type:" node=9
INF 2019-03-12 15:52:16+01:00 GossipSub Peer disconnected                tid=22243 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=9

Errors are in second and second-to-last line.
The node does not crash in this instance, it continues along after these errors are thrown.

Gossipsub: message filtering

With regards to ethereum/eth2.0-specs#593

When messages are received on the network, we should not propagate them unless they're "good" (to be defined).

libp2p allows blacklisting peers by peer id (which is secured by the above feature, https://github.com/libp2p/go-libp2p-pubsub/blob/master/blacklist_test.go#L38-L64).

There also exists a validation api (https://github.com/libp2p/go-libp2p-pubsub/blob/bfd65a2f6b810c5b4ad2cfe6bb9cc792fd7a0171/floodsub_test.go#L360) to filter messages by content - it is making its way into the daemon api as well: libp2p/go-libp2p-daemon#71

How do we make use of this information? should it affect peer scoring?

[Simulation] Change in BLS scheme #80 (?) breaks withdrawal_credentials

cc @zah

Change in BLS scheme #80 (?) broke the simulation in tests/simulation/start.sh.

Reproduced on Mac and Linux.

What is strange is that BLS passes all the tests including eth2.0-tests with the EF test generator and it also passes the classic beacon-chain tests.

beacon_node.nim(416)     beacon_node
trusted_state_snapshots.nim(33) createStateSnapshot
beaconstate.nim(214)     get_initial_beacon_state
beaconstate.nim(87)      process_deposit
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /Users/tesuji/Programming/Status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim(87, 12) `state.validator_registry[index].withdrawal_credentials ==
    withdrawal_credentials`  [AssertionError]
Terminated: 15

[Simulation] Receiving an invalid block can trigger segfault on stringifying a hash

NOT 2019-03-26 14:50:04+01:00 Invalid block                              tid=3075 blck="(slot: 7, previous_block_root: \"52C0B4F5\", state_root: \"06052C4E\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits
_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6d48dda\")" blockRoot=9547363C node=1                                                                                                                                            
Traceback (most recent call last)                                                                                                                                                                                                                 
beacon_node.nim(771)     beacon_node                                                                                                                                                                                                              
beacon_node.nim(691)     start                                                                                                                                                                                                                    
beacon_node.nim(664)     run                                                                                                                                                                                                                      
asyncloop.nim(808)       runForever                                                                                                                                                                                                               
asyncloop.nim(260)       poll                                                                                                                                                                                                                     
asyncmacro2.nim(36)      proposeBlock_continue                                                                                                                                                                                                    
chronicles.nim(167)      proposeBlockIter                                                                                                                                                                                                         
digest.nim(35)           shortLog                                                                                                                                                                                                                 
hash.nim                 $                                                                                                                                                                                                                        
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

as of https://github.com/status-im/nim-beacon-chain/commits/master

Pending attestation & block pool

During the call today, we discussed repair and compared it with sync. I mentioned that one way of approaching the problem is to split out unverified blocks and attestations into a separate space - the idea is similar to tainted strings or a firewalled network - basically, we would uphold stronger invariants in a "safe space" and make sure to guard the gate between the two..

Necessary criteria for entering the safe space would be at least the following (Item here is either an attestation or block):

the item is valid/consistent/sound, per all rules in the spec
the item has a fully resolved path to a finalized state we know about
- a stronger guarantee is that it has a path to the latest resolved path we know about - this is somewhat related to attestation and block pruning strategies
- this is necessary because without it, we can't establish that the signature is valid
the item has a valid signature by at least one known validator, as verified after resolving a path
- it must be signed by an existing validator so we can slash them in case the block is wrong
- the validator might have been ejected, hence resolved path is needed to properly validate signature

Another way to express those criteria more succinctly is that we should only permit items in there that have slashing conditions attached to them, in case they turn out to be malicious.

Pro:

simplifies code in the safe zone - we can assume that it's consistent and doesn't need further syncing
clearly delineates "useful" and "unknown" items

Con:

code inside safety zone relies on more assumptions being upheld -> becomes more fragile in face of bugs / issues
if something sneaks past safeguards, damage is high

AssertionError: `previous_epoch <= epoch`

Hit this after following these directions and running for a day or so.

Beacon Chain version:

~/src/status-im/nimbus/vendor/nim-beacon-chain $ git rev-parse HEAD
c9e0fed4eac43e1867465a88cf9d918b6dd8c8fe

Client Version:

~/src/status-im/nimbus $ git rev-parse HEAD
655a997559664b53424673b96f3818201f51c96d

rocksdb version:

$ brew info rocksdb
rocksdb: stable 5.18.3 (bottled)
Embeddable, persistent key-value store for fast storage
https://rocksdb.org/
/usr/local/Cellar/rocksdb/5.18.3 (107 files, 59.4MB) *
  Poured from bottle on 2019-03-29 at 21:08:32
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/rocksdb.rb
==> Dependencies
Required: gflags ✔, lz4 ✔, snappy ✔
==> Analytics
install: 1,278 (30 days), 3,499 (90 days), 21,878 (365 days)
install_on_request: 541 (30 days), 1,532 (90 days), 8,767 (365 days)
build_error: 0 (30 days)

Hardware:

Log:

len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD                                                                       [2/1824]
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_
len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Unresolved block                           tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5066, previous_block_root: \"A96F8DBD\", state_root: \"00094835\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"aa4b196c\")" blockRoot=3CD9C0C4
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5070, previous_block_root: \"167A5A61\", state_root: \"96DC41A7\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b444eb3a\")" blockRoot=41EA2013
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Unresolved block                           tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5056, previous_block_root: \"0FE1D1C3\", state_root: \"2B530C50\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"951cd66e\")" blockRoot=6DE1F47F
DBG 2019-03-31 06:50:03-07:00 Unresolved block                           tid=5348663 blck="(slot: 5056, previous_block_root: \"0FE1D1C3\", state_root: \"2B530C50\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"951cd66e\")" blockRoot=6DE1F47F
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5055, previous_block_root: \"35B06FD7\", state_root: \"F06CA02F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6b82eef\")" blockRoot=0FE1D1C3
DBG 2019-03-31 06:50:03-07:00 Unresolved block                           tid=5348663 blck="(slot: 5055, previous_block_root: \"35B06FD7\", state_root: \"F06CA02F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6b82eef\")" blockRoot=0FE1D1C3
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5074, previous_block_root: \"F2D2ABEC\", state_root: \"C0B2D833\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"855cbf22\")" blockRoot=BA067036
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5072, previous_block_root: \"9B5891B0\", state_root: \"CC15B79D\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b3bb91ef\")" blockRoot=F2D2ABEC
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5071, previous_block_root: \"41EA2013\", state_root: \"47651ED4\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b83ddb59\")" blockRoot=9B5891B0
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5070, previous_block_root: \"167A5A61\", state_root: \"96DC41A7\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b444eb3a\")" blockRoot=41EA2013
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5066, previous_block_root: \"A96F8DBD\", state_root: \"00094835\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"aa4b196c\")" blockRoot=3CD9C0C4
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5065, previous_block_root: \"70C2BCBB\", state_root: \"5912FAEA\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5065, previous_block_root: \"70C2BCBB\", state_root: \"5912FAEA\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received                             tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Slot start                                 tid=5348663 lastSlot=5525 scheduledSlot=5526 slot=5526
DBG 2019-03-31 06:50:03-07:00 Advancing state with empty slots           tid=5348663 stateSlot=5525 targetSlot=5526
Error: unhandled exception: /Users/nathanielgentile/src/status-im/nimbus/vendor/nim-beacon-chain/beacon_chain/spec/validator.nim(152, 12) `previous_epoch <= epoch` Previous epoch: 689, epoch: 688 (slot: 5507), Next epoch: 691 [AssertionError]

I don't have the entire log due to tmux history limit, but here is a more complete log: full-log

beaconnode createChain crashes with a strange error related to Cygwin in the current master https://github.com/status-im/nim-beacon-chain/tree/dda24619e824ef78c216d8a99ee280e57f182aa8 compared to the master from a week ago: https://github.com/status-im/nim-beacon-chain/tree/fix-sim-init-windows.

Full backtrace with debugging symbol:

#0  __chkstk_ms () at ../../../../../src/gcc-4.9.1/libgcc/config/i386/cygwin.S:172
#1  0x0061b2aa in get_initial_beacon_state_vi9cRmmSUyyCGR9cJE84pqSw (initial_validator_deposits=0x5f2c3165, initial_validator_depositsLen_0=1887007839, genesis_time=6857984371532455724, latest_eth1_data=0x14cbaa8, flags=116 't', Result=0x14cbae8) at D:/Devel/Status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim:159
#2  0x004a7f48 in createStateSnapshot_ahCWD5wcxfkrgWQSdv9cjfQ (startup=..., outFile=0x173a8e8) at D:/Devel/Status/nim-beacon-chain/beacon_chain/trusted_state_snapshots.nim:33
#3  0x00439e9d in NimMainModule () at D:/Devel/Status/nim-beacon-chain/beacon_chain/beacon_node.nim:402
#4  0x00439c11 in NimMainInner () at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3404
#5  0x00439c35 in NimMain () at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3412
#6  0x00439c5f in main (argc=4, args=0x1b914a8, env=0x1b91d98) at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3419

Note that this is with 32-bit Nim

AssertionError in get_crosslink_committees_at_slot after slot 64

On Linux x86-64

Similar to #89 #97 we have an issue during epoch transition, this times after 64 slots, for epoch 144115188075855873

Full logs available in https://gist.github.com/mratsim/a6539c049dfb49a638327b9db5cec44a.
There are 2 logs, one with multiple nodes desynced, one with only a single node desynced.

The following log is after merging #103 for epochs printing.

beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=338 slot=121 validator=96d7e599 node=0
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=296 slot=114 validator=8e724e86 node=5
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=326 slot=119 validator=0a89ffbe node=2
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=350 slot=123 validator=91792a76 node=7
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=344 slot=122 validator=842ef0e2 node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=356 slot=124 validator=861b6b0b node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=362 slot=125 validator=928a8a8e node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=374 slot=127 validator=9090c4be node=1
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=380 slot=128 validator=968a8996 node=8
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
Terminated