Problem: compact numeric byte arrays are encoded with 2 bytes about pumpkindb HOT 5 CLOSED

This reminds me of:

1. bitstream encodings https://en.wikipedia.org/wiki/Bitstream
2. schema-based encodings like protobuf/Cap'n Proto which compress small values to few bytes.

from pumpkindb.

Rinat, thanks for chiming in!

Right now, PumpkinScript's data representation is rather simple: https://github.com/PumpkinDB/PumpkinDB/blob/master/src/script/mod.rs#L115-L141

I am not particularly married to it but I find it just easy enough to actually use so far :) But I am very open to improvements.

I am aware of protobuf/Cap'n Proto and other approaches (like my own ELF/BEST — see https://rfc.eventsourcing.com), but here the situation is simple and (in a way) unique. There are only two items possible: byte arrays (ranging from 0-byte arrays to... lets say 2^32) and words (short byte arrays from 1 to 127 bytes long).

from pumpkindb.

Ah, thank you for the explanation. Now I better understand the rationale behind your encoding choices (simplicity and extensibility).

By the way, do you intend to leverage LMDB's capability of scanning across keys (to maintain event stream indexes or build projections operating on large sorted collections)?

from pumpkindb.

@abdullin scanning across keys is critical for indexes — which is a huge component in PumpkinDB

from pumpkindb.

This optimization is not worth it.

from pumpkindb.