plausiblelabs / rcodec Goto Github PK

When this library was first developed, procedural macros (as used by the underlying hlist crate) weren't yet stable, and there was a lot of boilerplate required for using macros from hlistand thisrcodec` crate.

We can clean the code up quite a bit by adopting Rust 2018 edition, removing now-unneeded extern crate, etc.

When we updated to Rust 2018 in #1, we had to temporarily disable the old benchmark-related code since there have been changes in that area in the past few years.

That code was commented out until we can address it separately with this issue.

Thanks for the crate. I really like the philosophy of invertible Codecs.

I ran into a problem parsing many small elements, of a stack overflow exception.

The test case below produces a stack overflow with only 8KB of data.

I believe that this is due to the fact that the library tries to avoid allocations by not consolidating arrays in ByteVector's and so they remain on the stack.

I think the fix would be to monitor the length of the ByteVector and when it goes above a threshold replace the current tree of ByteVectors of arrays (on the stack) with a single one which contains a Vec (on the heap).

Error:

thread 'codec::test::small_vec_repeat_should_work' has overflowed its stack
fatal runtime error: stack overflow

Test Case

  #[test]
    fn small_vec_repeat_should_work() {
        let mut b=byte_vector::empty();

        for i in 0..1000 {
            let small_array=[0u8;8];
            let small_vector=byte_vector::from_slice_copy(&small_array);
            b=byte_vector::append(&b,&small_vector);
            println!("len {}",b.length())
        }

        let vec=b.to_vec().unwrap();
        println!("Vec Len {:?}",vec.len())
    }

Workaround

I made a really nasty workaround of every time I append ByteVectors, I write a Vec of the result and then create a new ByteVector with that vector.