ralexstokes / ssz-rs Goto Github PK

View Code? Open in Web Editor NEW

83.0 83.0 32.0 2.18 MB

Implementation of ethereum's `ssz`

License: Apache License 2.0

Rust 99.57% Python 0.43%

ssz-rs's People

Contributors

Stargazers

Watchers

Forkers

snowfork rjected aga7hokakological mattsse ncitron wizdave97 jacobkaufmann polytope-labs 0xtylerholmes evalir

ssz-rs's Issues

Run against the latest tests from the `consensus-spec-tests`

Panic on deserialization

I was trying to write test cases for https://github.com/ralexstokes/ethereum_consensus to check if operations work correctly, but faced problems with the deserialization of BeaconBlock from block.ssz_snappy

Here is code for reproducing panic:

use ethereum_consensus::phase0::mainnet::BeaconBlock;
use project_root;
use snap;
use ssz_rs::prelude::*;
use std::fs::File;
use std::io::Read;
use std::path::Path;
use std::path::PathBuf;

#[test]
fn test_invalid_parent_root() {
    let path = Path::new("consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/invalid_parent_root/");

    let block_path = path.join("block.ssz_snappy");
    let block_path = block_path.to_str().expect("is valid path");

    let block_data = read_ssz_snappy_from_test_data(block_path);
    println!("block data from file {:?}", block_data);

    // for clarity
    let default_block = BeaconBlock::default();
    let ser_default_block = ssz_rs::serialize(&default_block).expect("can serialize");
    println!("default block data {:?}", ser_default_block);

    let block: BeaconBlock = ssz_rs::deserialize(&block_data).expect("can deserialize");
    println!("block from file {:?}", block);
}

// Return SSZ-encoded bytes from test file at `target_path`
pub fn read_ssz_snappy_from_test_data(target_path: &str) -> Vec<u8> {
    let project_root = project_root::get_project_root().unwrap();
    let target_path = PathBuf::from(target_path);
    let data_path = project_root.join(&target_path);
    let mut file = File::open(&data_path).expect("can read file");
    let mut data = vec![];
    let _ = file.read_to_end(&mut data).expect("can read file data");
    let mut decoder = snap::raw::Decoder::new();
    decoder
        .decompress_vec(&data)
        .expect("can decompress snappy")
}

Output:

block data from file [1, 0, 0, 0, 0, 0, 0, 0, 174, 0, 0, 0, 0, 0, 0, 0, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 132, 104, 188, 10, 245, 110, 113, 55, 111, 77, 65, 137, 20, 99, 112, 142, 241, 25, 41, 49, 219, 43, 36, 235, 53, 18, 216, 51, 83, 152, 0, 250, 94, 3, 9, 107, 173, 249, 30, 84, 252, 0, 232, 139, 156, 254, 184, 37, 18, 253, 41, 181, 18, 160, 148, 212, 215, 136, 104, 3, 185, 127, 64, 10, 147, 134, 151, 134, 219, 131, 237, 40, 151, 198, 107, 90, 44, 119, 157, 111, 151, 200, 255, 110, 221, 150, 109, 13, 144, 129, 255, 201, 55, 90, 18, 40, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0]
default block data [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 192, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0]
thread 'test_invalid_parent_root' panicked at 'range end index 4 out of range for slice of length 0', /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:50:42
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   2: core::slice::index::slice_end_index_len_fail
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:41:5
   3: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:240:13
   4: <core::ops::range::RangeTo<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:286:9
   5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:15:9
   6: ssz_rs::de::deserialize_variable_homogeneous_composite
             at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:50:42
   7: ssz_rs::de::deserialize_homogeneous_composite
             at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:78:9
   8: <ssz_rs::list::List<T,_> as ssz_rs::de::Deserialize>::deserialize
             at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/list.rs:142:22
   9: ethereum_consensus::phase0::beacon_block::BeaconBlockBody<_,_,_,_,_,_>::__ssz_rs_set_by_index
             at ./src/phase0/beacon_block.rs:21:5
  10: <ethereum_consensus::phase0::beacon_block::BeaconBlockBody<_,_,_,_,_,_> as ssz_rs::de::Deserialize>::deserialize
             at ./src/phase0/beacon_block.rs:8:26
  11: ethereum_consensus::phase0::beacon_block::BeaconBlock<_,_,_,_,_,_>::__ssz_rs_set_by_index
             at ./src/phase0/beacon_block.rs:41:5
  12: <ethereum_consensus::phase0::beacon_block::BeaconBlock<_,_,_,_,_,_> as ssz_rs::de::Deserialize>::deserialize
             at ./src/phase0/beacon_block.rs:28:26
  13: ssz_rs::deserialize
             at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/lib.rs:62:5
  14: operations_tests::test_invalid_parent_root
             at ./tests/operations_tests.rs:41:30
  15: operations_tests::test_invalid_parent_root::{{closure}}
             at ./tests/operations_tests.rs:27:1
  16: core::ops::function::FnOnce::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
  17: core::ops::function::FnOnce::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5

Same panic if I try to deserialize serialized BeaconBlock::default()

Panic while computing `hash_tree_root` of `BeaconState`

Trying to compute hash_tree_root of BeaconState deserialized from "consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/success_block_header/post.ssz_snappy" results in panic:

thread 'phase0::state_transition::block_processing::tests::test_process_block_header' panicked at 'range end index 32 out of range for slice of length 0', C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:182:17
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\std\src\panicking.rs:517
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\core\src\panicking.rs:100
   2: core::slice::index::slice_end_index_len_fail
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\core\src\slice\index.rs:41
   3: core::slice::index::impl$3::index_mut<u8>
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\slice\index.rs:251
   4: core::slice::index::impl$1::index_mut<u8,core::ops::range::Range<usize> >
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\slice\index.rs:26
   5: alloc::vec::impl$17::index_mut<u8,core::ops::range::Range<usize>,alloc::alloc::Global>
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\alloc\src\vec\mod.rs:2508
   6: ssz_rs::list::List<ethereum_consensus::phase0::validator::Validator,1099511627776>::compute_hash_tree_root<ethereum_consensus::phase0::validator::Validator,1099511627776>
             at C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:182
   7: ssz_rs::list::impl$12::hash_tree_root<ethereum_consensus::phase0::validator::Validator,1099511627776>
             at C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:248
   8: ethereum_consensus::phase0::beacon_state::impl$14::hash_tree_root<8192,16777216,2048,1099511627776,65536,8192,2048,4096>
             at .\src\phase0\beacon_state.rs:64
   9: ethereum_consensus::phase0::state_transition::block_processing::tests::impl$0::execute
             at .\src\phase0\state_transition\block_processing.rs:834
  10: ethereum_consensus::phase0::state_transition::block_processing::tests::impl$0::execute_from_glob
             at .\src\phase0\state_transition\block_processing.rs:865
  11: ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header
             at .\src\phase0\state_transition\block_processing.rs:872
  12: ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header::closure$0
             at .\src\phase0\state_transition\block_processing.rs:871
  13: core::ops::function::FnOnce::call_once<ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header::closure$0,tuple$<> >
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\ops\function.rs:227
  14: core::ops::function::FnOnce::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Code for reproduction:

    fn test_panic() {
        let path = "consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/success_block_header/post.ssz_snappy";
        let post_state = read_ssz_snappy_from_test_data(&path).expect("file exists");
        let mut post_state: BeaconState = deserialize(&post_state).expect("can deserialize");
        let result = post_state.hash_tree_root();
        println!("{:?}", result);
    }

unify handling of bytes

across this repo and https://github.com/ralexstokes/ethereum-consensus

there are many bytes-like types that have specific ways of printing/serializing/deserializing

let's unify across both repos

enhance indexing and arithmetic safety

analogues of these:

ralexstokes/ethereum-consensus#151

ralexstokes/ethereum-consensus#152

for this repo :)

Refactor test generation

A series of tasks to improve the test gen, which reads the test artifacts to make a set of Rust "harnesses" which consume the data to exercise the implementation.

Clearly define which set of test vectors we use (iirc only part of the data is needed)
Add some convenience commands to manage the data, like in this justfile https://github.com/ralexstokes/ethereum-consensus/blob/main/justfile
[bigger task] write a separate crate for a "testgen" utility: right now, this is done in python, but it would be nicer to bring everything under one (rust) toolchain. this should be a pretty straightforward rewrite of the python in rust, with the caveat that the utility should be in a separate crate so development in the "main" crate doesn't cause any embedded utility to fail to compile

Add blanket implementation for {mut-,} references

Add a blanket implementation for &T such that if T: SimpleSerialize then &T also implements SimpleSerialize.

And extend to &mut T as well unless there is an obvious reason not to do so.

Raise an error on incorrectly formatted hex

impls like this: https://github.com/ralexstokes/ssz_rs/blob/main/ssz_rs/src/bitlist.rs#L33-L47

do not verify there is a leading 0x and that can lead to accidentally skipping the first byte of data

look into fixing with trim_start_matches https://doc.rust-lang.org/std/string/struct.String.html#method.trim_start_matches and return an error if there is no leading 0x

rework "private module" for proc macro crate

investigate code coverage

some errors where silently dropped while working on this PR #25 (comment)

and the CI didn't raise any issue...

we should set up some code coverage and strive for 100%

harden "entrypoints" to SSZ layer

there are several ways to construct valid SSZ objects, e.g. from deserializing some bytes, or using a FromIterator implementation.

let's make sure that any way to consume or produce bytes from this library only allows for valid SSZ. in particular, this means making sure there is no Rust facility to construct "invalid" instances

update readme to explain how to run `no-std`

Do we have any idea to achieve a `no-std` feature of ssz-rs?

no-std fork stack overflow fatal errors on some tests

Not really your problem, but would appreciate some 👁️ on a stack overflow error that occurs when running tests that expects a panic.

support some kind of caching w/ the "Merkle backing" for `Merkleized` types

relevant data to consider:

ralexstokes/mev-rs#27

overflow when compiling for 32 bit architectures

When compiling to wasm32-unknown-unknown, this line overflows, since 2^32 cannot fit inside of usize. I hacked my way around this by just hard coding it to 2^32-1, but I imagine that is going to break something elsewhere. Guessing the better move would be to convert to u64.

support Merkle multi proofs

support proving and verification of Merkle (multi)proofs

add `merkle` handler for the `consensus-spec-tests`

merkle

refine CI

https://github.com/ralexstokes/ssz-rs/pull/25/files#r1029618826

add API for initializing a new `List`

Issue raised in comment from ethereum-consensus

Idea is to avoid something like the below and allow for a new List to be initialized without allocating a Vec and then converting into a List.

let rotate_participation = vec![ParticipationFlags::default(); state.validators.len()];
    state.current_epoch_participation = rotate_participation
        .try_into()
        .expect("should convert from Vec to List");

For example, add impl of new for something like List<T>::new(capacity: usize) (or, in the example above, just allow for state.current_epoch_participation = List::<u8>::new(state.validators.len()).

Based on some initial research, I'm not quite sure how to do this since List obviously takes List<T, N> where N is a const usize...so List<T> would be invalid. And without knowing N at compile time, that introduces concepts like raw pointers, unsafe rust, Box, Arc, etc. that I couldn't figure out how to use or if they even solve this problem.