TBD: https://github.com/ethereum/eth2.0-specs/blob/master/specs/casper_sharding_v2.1.md#state-recalculations-every-cycle_length-blocks

Description

I have created a test for BooleanBitfield that I think should pass, but fails.

bedc1ab

Present Behaviour

Test fails

Expected Behaviour

I think this test should pass

Steps to resolve

I haven't looked into how to resolve it.

Description

BeaconBlock has been split into BeaconBlock and BeaconBlockBody.

The code base must be updated to suit.

https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beacon-chain-blocks

Description

The spec has update and AttestationRecord needs to be updated.

This will likely be a fairly significant refactor as the AttestationRecord validation will need to be upated.

Randao for new epoch is not correctly generated. Implement the same (incomplete) method used in the reference impl:

https://github.com/ethereum/beacon_chain/blob/509387c5a1eb4eaa9f636ae01dedf4b340c4f09e/beacon_chain/state/state_transition.py#L439-L441

Description

SSZ tree hashing is not yet implemented, therefore we have had to "stub out" some of the canonical root methods.

Present Behaviour

The following states have the canonical_root method "stubbed out":

• AttestationData

Expected Behaviour

canonical_root() should return the canonical hash of an object. In all cases so far, this should be the SSZ tree hash (aka merkle root).

Steps to resolve

1. Resolve #70
2. Update canonical_root methods on all necessary functions.

We have abstracted the database into a ClientDB trait and implemented a DiskDB using RocksDB for production.

It would be useful to have an implementation of an in-memory database that can be used in testing. The benefits would be speed (i.e., RAM is faster than disk) and convenience (we don't need to touch the filesystem).

The in-memory database doesn't need to be particularly optimized or elegant, just the fact it's in memory should make it fast enough. I would suggest a two-layer nested hashmap to simulate columns. E.g., HashMap<HashMap<Vec<u8>>>. It will need to implement some Mutex/RwLock internally so it can be sent around threads like RocksDB can be.

It should live in a memory_db.rs file here and I think the struct should be called MemoryDB.

Description

Implement a test suite for the BeaconChain struct.

Present Behaviour

Presently there are no end-to-end tests for the BeaconChain struct. It's very important that this piece is heavily tested.

Shuffle needs the following work:

• It's still using the old blake2s hashing, which makes it incorrect.
• It should have some more test vectors.
• It should be separated into its own crate

Items in-scope of this task:

• Receiving some message from the network
• Determining that message is sending blocks
• Validate the block (according to the "Initial Block Validation" diagram)

Items specifically out-of-scope:

• Doing a state transition on the block.
• Verifying the attestation records.

Presently the code-base is missing the following features for BLS signatures and public keys:

1. Support for aggregate pub keys. Presently there is support for aggregate signatures, but not aggregate public keys, meaning each pubkey needs to be individually checked against the aggregate sig.
2. Ability to be cloned (i.e., they don't have the Clone trait)
3. Ability to be encoded to some primitive then have that be decoded back to a proper struct. For example, writing to a byte-array/integer then reading back again.
4. Closely related to (and dependent on) (2), they need to support RLP encoding and decoding. This probably requires the EF researchers to pick a format for this.

With regards to (1), I'm not even certain if aggregating the keys then verifying them even gives a better result. I.e., is it actually quicker to add them all together then verify once, or is it quicker to just verify them all individually? I'm presently assuming the latter is quicker but I don't know for sure.

Underflow impossible due to if list.is_empty() check above. Remove saturating_sub().

Problem

This concern shouldn't be enforced at this stage, SszBlock should be general enough to handle zero attestation records.

Required Fix

Remove the requirement that the SSZ must be long enough to accommodate one attestation record.

Description

Seems like we are missing a check on the incoming block data during validation.

We want to make sure that the block's ancestor_hashes align with our expectations given our local view of the chain.

Present Behaviour

We deserialized ancestor_hashes and just accept the data as given.

Expected Behaviour

The block should be invalid if we don't derive the expected updates to the ancestor_hashes.

Steps to resolve

We should check that every block updates the ancestor_hashes according to spec:

copied here for convenience (double check w/ latest spec before implementing...):

"
Also, check that the block's ancestor_hashes array was correctly updated, using the following algorithm:

def update_ancestor_hashes(parent_ancestor_hashes: List[Hash32],
                           parent_slot_number: int,
                           parent_hash: Hash32) -> List[Hash32]:
    new_ancestor_hashes = copy.copy(parent_ancestor_hashes)
    for i in range(32):
        if parent_slot_number % 2**i == 0:
            new_ancestor_hashes[i] = parent_hash
    return new_ancestor_hashes

"

Description here:

ethereum/consensus-specs#12

Description

It looks like this module is not being used. It does provide duplicate definitions of AttesterMap and ProposerMap which is confusing. It seems like this module is just dangling after a refactor.

Present Behaviour

it exists.

Expected Behaviour

this module should not exist if it is not being used.

Steps to resolve

Let's just remove it.

Opening this issue to confirm my suspicion that it is just dangling after a refactor. Otherwise just close this :)

Description

The on-boarding docs should be updated as we learn from gaining new contributors.

This issue can serve as a list of TODOs for items to add to the onboarding docs:

1. Every TODO must be accompanied by a link to a Github issue. This is to assist with project management.
2. Explain why we almost always don't use panic.
3. Point out that we use /// to generate doc comments.

Description

The BeaconChain struct is still using CrystallizedState or ActiveState.

Present Behaviour

Uses old structs.

Expected Behaviour

Not use old structs.

Steps to resolve

Refactor!

Description

ValidatorRecord has been outdated. Update it and all dependencies.

validator_induction is a definite dependency and it also needs to be updated to reflect the new spec definition. To do this, you'll need the presently-undefined Deposit and DepositParameters structs.

Tasks

• Update the ValidatorRecord struct.
• Add the Deposit and DepositParameters structs.
• Remove the types::validator_registration::ValidatorRegistration struct.
• Update the validator_induction component.
• Update any other components that are affected (feel free to ping me if you have any questions around this, it might get tricky).

Implement a generic "wrapper" trait so we can use a proper, persistent db (e.g., Rocks) during prod and a fast, transient db (e.g., memory) during testing.

Implement our own version of the types that are currently pulled from ethereum_types removing the create as a dependency.

Description

This issue here details an issue with the spec:

ethereum/consensus-specs#224

Present Behaviour

We have implemented a method that doesn't underflow or accept blocks until the min_attestation_inclusion_delay'th block.

Expected Behaviour

We must match the spec, but the spec is wrong IMO.

Steps to resolve

Wait until this issue is resolved, then either close this issue or update our code.

Description

Significant spec updates have required a re-write of block processing.

All existing block processing code has been removed and stubbed out.

To Resolve

Re-implement block processing

We presently have a ssz crate which uses the wrong type of serialization. It follows the ethereum/research/py_ssz scheme instead of the ethereum/beacon_chain scheme.

Also, it doesn't decode.

It should follow the beacon_chain serialization format and support encoding and decoding.

We only need it for Blocks and AttestationRecords at the moment, so it would be ideal to prioritize them.

Implement the add validator routine as defined: https://github.com/ethereum/eth2.0-specs/blob/master/specs/casper_sharding_v2.1.md#routine-for-adding-a-validator

As detailed here: https://ethresear.ch/t/blob-serialisation/1705

There are some basic implementations of "stores" here:

https://github.com/sigp/lighthouse/tree/master/lighthouse/db/stores

The goal of the stores concept is to abstract database operations away from "higher-level" parts of the application. Each store is initialized with some underlying database implementing the ClientDB trait. The store provides read/write access to specific parts of the database. Stores are separated by topic (e.g., blocks, validations) for two main reasons (a) it's nice to separate things and have smaller files and, (b) having separate stores helps to communicate which parts of the database some function might access to.

In the near future, stores may implementing caching, bloom filters and other whiz-bang things to speed up data access.

The ClientDB trait has a concept of "columns" to help separate the key space between stores. It's up to the underlying database as to how it implements these columns. MemoryDB just adds a prefix to each key in order to separate columns. DiskDB (RocksDB) might choose to implement some columns as actual RocksDB columns and others as just prefixes in the same physical column.

Presently, the tests for the stores are pretty dismal. It would be great to have some tests comprehensive tests. We only need to test against MemoryDB, using DiskDB is cumbersome and the two should always perform identically -- if they don't it's an issue for the databases ClientDB implementation.

Happy to assist anyone who wants to work on this :)

The ssz crate has been updated so it follows the beacon_chain version, however the README has not been updated to describe it.

Add a README.md which describes:

1. An overview of the crate.
2. An overview of the SSZ serialization method.
3. How to encode something.
4. How to decode something.
5. How to implement your own Encode and Decode traits.

https://github.com/sigp/rust_beacon_chain/blob/6f0bbd47fac20652472c1ff5799aac367af5d929/src/state/transition/attestations.rs#L30-L37

This is blocked by #1.

There is an implementation of a Boolean Bitfield here:

https://github.com/sigp/lighthouse/tree/master/boolean-bitfield

It (kinda) does the job for now, but it really needs some work done. If you spend some time looking at it I think you'll soon find out what I mean. As an example;

• There is a possibility of overflows: we return the number of bits as a usize, however there can theoretically be usize number of bytes meaning we can have 8 * usize bits.
• It keeps track of the number of true bits as you flip bits on and off. I don't think this is ideal as most cases where we want to know the number of true bits, we'll be receiving some serialized bytes from somewhere else (e.g., p2p nodes) and will need to calculate it manually.

On top of these two points, there's likely many chances for optimization.

Required Functionality

Get

get(n: usize) -> Result<bool, Error>

Get value at index n.

Error if bit out-of-bounds (OOB) of underlying bytes.

Set

set(n: usize, val: bool) -> Result<(bool, Error>

Set bit at index n. Returns the previous value if successful.

Error if bit is OOB of underlying bytes.

Highest Set Bit

highest_set_bit() -> Option<usize>

Returns the index of the highest set bit. Some(n) if a bit set set, None otherwise.

Note: this is useful because we need to reject messages if an unnecessary bit is set (e.g. if there are 10 voters and the 11th bit is set

Number of Underlying Bytes

num_bytes() -> usize

Returns the length of the underlying bytes.

_Note: useful to reject bitfields that are larger than required (e.g., if there are eight voters and two bytes -- only one byte is necessary). _

Number of Set Bits

num_set_bits() -> usize

Returns the total number of set bits (i.e., how many peeps voted).

Note: I'm not 100% sure we'll use this but I suspect we will.

Presently, SSZ prefixes a Hash32 with a length prefix. It should not do this, as a Hash32 is fixed-length and does not need a length prefix.

When this is implemented, it will break the SszBlock struct. I am more than happy to fix that struct personally when the time comes.

Add subtasks.

In particular this requires #11

Description

What's the only thing that changes more than Australian prime ministers?
....
The Eth 2.0 spec!!! lololol

The BeaconState object has changed and we need to update!

Description

The shuffle() function in the Eth 2.0 spec has changed. We need to update our function.

Resources:

• Our shuffle crate: https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/shuffling
• Spec update PR: ethereum/consensus-specs#65

Present Behaviour

RAND_MAX is presently set to 2**24.

Expected Behaviour

RAND_MAX should be 2**24 - 1.

Steps to resolve

1. Change RAND_MAX
2. Use sigp/shuffling_sandbox to generate some test vectors. There's only 1 test vector in there now, we should throw in multiple (make sure you use different list sizes).

Hint: shuffling_sandbox cmd: $ python sandbox.py print --list-size LIST_SIZE

Description

Attestation validation has been implemented, however the tests are not.

The tests rely upon the new block structure (#96) which has not been implemented.

Steps to resolve

Once #96 is complete, come back an implement tests.

Description

My personal rustfmt extension failed and a lot of the code I wrote is not formatted using rustfmt. I would like to run it across the entire code base.

Present Behaviour

Code is not formatted using rustfmt.

Expected Behaviour

Code should be formatted using rustfmt.

Steps to resolve

1. Halt as much development as possible.
2. Run rustfmt on the entire codebase

I will manage this myself, I don't need help on this one :)

Description

The spec declares that BLSVerify should take a "domain".

Present Behaviour

Our code base has no concept of a domain.

Expected Behaviour

Match the spec.

Steps to resolve

I am not aware how domain is implemented, this is a placeholder issue as a TODO was created in the code.

1. Figure out what a domain is in a BLS signature verification.
2. Implement it.

Presently we do not serialize the aggregate_sig field on an AttestationRecord.

We need to:

1. Figure out some way to sensibly serialize the value.
2. Check this method with the EF researchers (via the ethereum/sharding gitter)
3. Implement serializing the aggregate_sig on the AttestationRecord struct.

Description

Update types::Block to bring it inline with the latest spec updates.

Present Behaviour

Presently, we're using the block struct from several spec revisions ago.

Expected Behaviour

Be up-to-date

Steps to resolve

1. Change Block -> BeaconBlock.
2. Add/remove fields as per new spec.
3. Update SszBlock.
4. Update all dependant functions.

Currently validator delegation, more specifically, generate_cycle() in transition/delegation/validator.rs could do with extra logic in the tests.

Specifically checking expected committee sizes and more tests checking edge cases.

Description

Implement the function described here: ethereum/consensus-specs#54

Present Behaviour

Function doesn't exist.

Expected Behaviour

Function should exist.

Steps to resolve

AFAIK, the function is still experimental so be on the lookout for bugs and optimisations.

At this stage, I think we should implement it as a separate crate.

Description

There are scenarios in attestation_validation::validate_attestation_signature that are untested.

You can find the crate at /beacon_chain/attestation_validation (once #85 has been merged into master).

Here is a non-comprehensive list to get you started:

• No signatures at all (aggregate signature is empty/at infinity).
• There is an extra signature on the attestation.
• The database fails.
• A validator index is unknown.

Steps to resolve

1. Determine all code paths in the validate_attestation_signature function (within reason -- ask here for canonical definition of "reason")
2. Implement tests for reasonable code paths.

TBD: https://github.com/ethereum/eth2.0-specs/blob/master/specs/casper_sharding_v2.1.md#dynasty-transition

Description

It would be great to have a guide that directs users on how to install Rust and run the test suite. For example:

1. Obtain and install cargo (linux, mac and windows)
2. Use cargo to install stable
3. Use stable to run the test suite (e.g., cargo test --all)

I recommend giving high-level steps that an experienced developer could follow, then linking to more detailed articles that provide "hand holding".

!! WAITING ON RELEASE CANDIDATE !!

This issue is presently on-hold until the specification is stable.

Description

This is a "mega issue" which contains multiple discreet tasks. Mega issues exist to avoid cluttering the issues page a multitude of small tasks.

If you wish to work on one of these tasks, comment below and a maintainer will break the task out into a separate issue.

Mega Issue Tasks

The definition for all structs can be found in the Data Structures section of the specification.

Types are to be defined in separate files in the lighthouse/beacon_chain/types crate here.

All of these tasks should be quite simple.

• Update the ShardAndCommittee struct.
• Update the SlashableVoteData struct.
• Update the AttestationData struct.
• Add the AttestationDataAndCustodyBit struct.
• Update the DepositInput struct.
• Update the BeaconBlock struct.
• Update the BeaconBlockBody struct.
• Update the BeaconState struct.
• Update the ValidatorRecord struct. Included in #116.
• Update the CrosslinkRecord struct.
• Update the ShardCommittee struct.
• Add the DepositRootVote struct.
• Add the ValidatorRegistryDeltaBlock struct.

Description

The clippy linter has a lot to say about our ssz implementation. We'd like to remove these warnings so we can eventually start failing builds if they trigger clippy.

Present Behaviour

Clippy outputs warnings when run against the ssz crate.

Expected Behaviour

Clippy should not have any warnings.

Steps to resolve

1. Run clippy
2. Fix lints

Bonus points

• Tidy up the code (there's definitely some cases where things could be neater)
• Optimise the code

The crate was written during an experimental phase of SSZ and is not as elegant as we would like. We're not attached to it and would be open to suggestions on a restructure. Please make a new issue if you'd like to restructure :)

TODO: fill out this description

Description

The spec updated and the CrystallizedState and ActiveState are now merged into the one object.

Present Behaviour

We have two, distinct states.

Expected Behaviour

The states should be unified

Description

The genesis_states function in the chain crate has been stubbed out as new updates to the spec have made it mostly redundant.

Steps to resolve

Implement the new genesis code after #96 has been merged.