Prototype implementation of zarr v3 in rust.
Based heavily on an earlier prototype at sci-rs/zarr.
See examples/roundtrip.rs
for an example (and cargo run --example roundtrip
to run it).
Prototype implementation of zarr v3 in rust
License: MIT License
Prototype implementation of zarr v3 in rust.
Based heavily on an earlier prototype at sci-rs/zarr.
See examples/roundtrip.rs
for an example (and cargo run --example roundtrip
to run it).
Currently a mess of unwrap and Result<_, &'static str>
.
This will mean changing a bunch of signatures, and probably revisiting the endian codec's handling, using the fallible write_whatever::<BigEndian>
rather than the panicking BigEndian::write_whatever
which wasn't working before.
Currently, ArrayMetadata cannot be generic over the data type because it needs to deserialise the fill_value
based on the data_type
. The ArrayMetadataBuilder
and the Array
are both generic, so the concrete metadata is a weird in-between. This means that the array knows what kind of values it needs but can't communicate it to the type system: the best we can do is be generic over T: ReflectedType
and then do a runtime check like if T::ZARR_TYPE != self.data_type { panic!("oops") }
.
Presumably after addition of checksum.
Preferably, codecs involving bytes (BB, AB) would use Readers/ Writers instead of allocating new byte buffers at every stage. This is how they were originally implemented ( a2757db ). However, with the introduction of the CRC32C codec (and conceptually other hashsums), decoders may need to read the whole read stream (to remove the hash). This is not too much of a problem - it can read the whole stream, then present a Boxed Reader which just wraps over the read bytes. However, encoders may need to know when the write stream is finished (to append the hash). We could:
Drop
to write the hash, but this can't be guaranteed, especially as it's fallibleflush()
by assuming it will only be called at the end of the write, but that's a big assumption.finalize()
method which is a no-op for most writers: this could require different wrappers for different codecs, and we'd still need to pass around something like Box<CustomWriter<Box<dyn Write>>>
Bytes
objects; for decoding we might get lucky and be able to use slices of them.4 is the simplest and brings us closer in line with other implementations. Allocation reduction is probably going to be dwarfed by IO and encode/decode overhead.
"NaN"
"+Infinity"
"-Infinity"
"0xYYYYYYYY"
for specific NaNsAlso for complex.
In accordance with zarr-developers/zarr-specs#237
This may add some complications to lazy-reading the footer, but then again it's small enough that just doing it eagerly and returning a Bytes should be fine.
Codecs which currently return boxed readers and writers could use an associated type on the trait instead. Boxed dyn trait objects would probably still be needed for e.g. impl ABCodec for CodecChain
and impl BBCodec for &[BBCodec]
, but it could reduce the number of required boxes. Boxed objects may also still be necessary for partial IO
(en|de)code_into
Where arrays need to be passed around, the output array could also be given (as a mutable view). This would itself be the decoded representation, and reduce allocations/ clones.
(en|de)code
We have partial IO at the store level; codec level is necessary to make sharding worthwhile.
Obviously a lot goes on at the same time in this context. I'd lean towards async-std if using async/await as it seems more library-friendly. But the lifetime-wrangling involved makes me nervous.
The minimal version would probably be to make the chunk IO methods on Array async but leave the codecs synchronous, then allow region reads/writes to just join over the internal futures (possibly blocking on that so the function itself doesn't need to be async).
Slices, smallvecs, &smallvecs, vecs; u64 or usize.
ndarray
just panics if mismatched dimensions are given when working with IxDyn
; can probably follow that way.
Prevents runtime errors for sharding codec creation, but doesn't protect against reading illegal sharding codecs. Maybe don't need to worry about that?
Partial reads are going to be a pain, but it would be good to have complete reads/writes for the current sharding spec available with well-documented tradeoffs.
Probably needs a CodecError enum which contains an "Other" type for potential future codec-specific errors. This could roll io::Error
into it.
https://crates.io/crates/object_store
get
and get_range{s}
use an async stream wrapping a Bytes
With any move to async, BB codecs in particular would need major rewrites, and make use of https://crates.io/crates/async-compression for compression.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.