cswinter / compressed-vec Goto Github PK

Floating point and integer compressed vector library, SIMD-enabled for fast processing/iteration over compressed representations.

This is a compressed vec library, rather than a compression library. What does that mean? A compression library takes some uncompressed data and provides essentially compress() and decompress() functions. Typically you have to decompress data to be able to do anything with it, resulting in extra latency and allocations.

On the other hand, this compressed vec library allows you to iterate over and process the compressed representations directly. It is designed to balance fast iteration and SIMD processing/filtering, while compressing vectors to within 2x of the best columnar compression technology such as Apache Parquet, using techniques such as delta and XOR encoding. Applications:

• Database engines
• Large in-memory data processing
• Games and other apps needing fast access to large quantities of FP vectors/matrices

Numbers are from my laptop: 2.9 GHz Core i9, 6/12 cores, 12MB L3, AVX2; from running cargo bench vector, which benchmarks a filter-and-count-matches operation directly on encoded/compressed vectors.

• The two u32 vector filtering speed (using MultiVectorFilter) depends on the order of the vectors. It is faster to filter the sparse vector first.

To create an f32 compressed vector:

    use compressed_vec::VectorF32XorAppender;
    let mut appender = VectorF32XorAppender::try_new(2048).unwrap();
    let encoded_bytes = appender.encode_all(vec![1.0, 1.5, 2.0, 2.5]).unwrap();

The simplest way to iterate on this compressed vector (note this does not allocate at all):

    use compressed_vec::VectorReader;
    let reader = VectorReader::<f32>::try_new(&encoded_bytes[..]).unwrap();
    let sum = reader.iterate().sum::<f32>();   // Yay, no allocations!

iterate() is the easiest API to go through individual elements of the compressed vector, but it is not the fastest. Fast data processing, such as done in the filter-and-count benchmarks above, are performed using Sinks, which are defined in the sink module. Sinks operate on a SIMD word at a time, and the sink API is designed for inlining.

For example, let's say that we want to add 2.5 to the f32 vector above, and then write out the results to a Vec<f32>. Internally, XOR encoding and decoding is performed (using a sink). The sinks can be stacked during decoding, for an almost entirely SIMD pipeline: - XorSink (used automatically for f32 decoding) - AddConstSink (to add 2.5, again done using SIMD) - VecSink (writes output to a normal Vec)

    use compressed_vec::{VectorReader, AddConstSink, VecSink};
    let reader = VectorReader::<f32>::try_new(&encoded_bytes[..]).unwrap();
    let mut vecsink = VecSink::<f32>::new();
    let mut addsink = AddConstSink::new(2.5f32, &mut vecsink);
    reader.decode_to_sink(&mut addsink).unwrap();
    println!("And the transformed vector is: {:?}", vecsink.vec);

Details of the vector format can be found here.

The vector format follows columnar compression techniques used throughout the big data and database world, and roughly follows the Google Procella paper with its custom Artus format:

• Compression within 2x of ZSTD while operating directly on the data
  • Compression for this format is within 2x of Parquet, but is written to allow filtering and operating on the data directly without needing a separate decompression step for the entire vector
• Multi-pass encoding
  • The VectorAppender collects min/max and other stats on the raw data and uses it to decide on the best encoding strategy (delta, etc.)
• Exposing dictionary indices to query engine and aggressive pushdown to the data format
  • The format is designed to filter over dictionary codes, which speeds up filtering
  • The use of sections allows for many optimizations for filtering. For example, null sections and constant sections allow for very fast filter short-circuiting.

Please reach out to me to collaborate!