serde-rs / bytes Goto Github PK
View Code? Open in Web Editor NEWWrapper types to enable optimized handling of &[u8] and Vec<u8>
License: Apache License 2.0
Wrapper types to enable optimized handling of &[u8] and Vec<u8>
License: Apache License 2.0
I'm trying to make some structures serialize into bytes in a nice way. However, these types contain both optional data and maps of bytes (Option<Vec<u8>>
and BTreeMap<usize, Vec<u8>>
). #[serde(with = "serde_bytes")]
doesn't seem capable of dealing with these types. I've found a way to make them work by changing the types to Option<ByteBuf>
and BTreeMap<usize, ByteBuf>
, but it adds quite a lot of extra glue code. Is there a nicer way to achieve this?
Sorry, I am still a new Rust user. I realize it's a wrapper, but I can't seem to convert pieces to an Vec!!!
pieces: b"J\u{a0}\u{c3}\u{fb}\u{ce}q&\u{8c}N\u{fb}V\u{fe}L\u{b1}\u{f4}\"&\u{bc}YY&\u{c5}\u{f5}\u{90}\u{f8}\u{8d}\u{c5}`\u{90}]-,\u{1d}J\u{82}\u{db}9O\u{ae}\u{98}\u{c4}Sa\u{a1}\u{85}\u{8c}7\u{cf}\u{df}w\u{bb}qlH\u{fa}6\u{8f}6\u{5}\u{af}MB\u{89}\u{c7}i\u{94}\u{ee}\u{95}\u{b0}0+v\u{ca}\r\u{f2}\u{a3}Q\u{a1}\n\u{fc}\u{84}\u{ea}\u{c8}-?8>l\u{1b}\u{b9}\u{d5}\u{a0}\u{c1}\u{8b}\\\u{db}\u{c1}\u{b7})\u{af}\u{db}&_\u{87}\u{a7}\u{f6}\u{4}y\u{16}\u{c3}\u{2}\u{98}G\u{9c}\u{ae}\u{3}\u{c9}\u{dc}\u{ec}\u{cb}\u{fa}(W\u{f4}\u{fb}\u{eb}M>\u{9d}}\u{84}~K\u{94}\u{c6}\u{b4}\u{18}\u{f4}\u{fa}\u{83}"
This is my current self.info.pieces
Hi,
I'm having a hard time figuring out how exactly #[serde(with = "serde_bytes")]
on a &[u8]
provides any efficiency gains for deserialization. From what I can see, this causes serde_bytes::deserialize()
to be called, which forwards to serde_bytes::Deserialize for &[u8]
, which just calls serde::Deserialize::deserialize()
. How is this any different from not using serde_bytes
at all? Am I missing something, or is this crate simply useless for optimizing deserialization of &[u8]
?
Currently Cow<'a, [u8]>
uses &'a [u8]
deserialization implementation which would fail when deserializing from any non-binary format.
While this is expected for &'a [u8]
but Cow<'a, [u8]>
can contain Vec<u8>
as well and doesn't have to be a pointer into deserializer owned data.
I suggest to write a special visitor for Cow<'a, [u8]>
that will create Cow::Borrowed
when visit_borrowed_bytes
or visit_borrowed_str
is called, and Cow::Owned
for visit_str
, visit_string
, visit_bytes
or visit_byte_buf
.
Hello! Thanks for this great crate! I tried to add Vec<Vec<u8>>
under efficient bytes conversion but was unable to. Probably expected, would it be possible to add such support?
More detailed errors below:
...
error[E0277]: the trait bound std::vec::Vec<std::vec::Vec<u8>>: std::convert::AsRef<[u8]>
is not satisfied
--> src/lib.rs:169:10
|
169 | #[derive(Serialize, Deserialize, Debug, Clone)]
| ^^^^^^^^^ the trait std::convert::AsRef<[u8]>
is not implemented for std::vec::Vec<std::vec::Vec<u8>>
|
= help: the following implementations were found:
<std::vec::Vec as std::convert::AsRef<[T]>>
<std::vec::Vec as std::convert::AsRef<std::vec::Vec>>
= note: required by serde_bytes::serialize
error[E0277]: the trait bound std::vec::Vec<std::vec::Vec<u8>>: std::convert::From<std::vec::Vec<u8>>
is not satisfied
--> src/lib.rs:169:21
|
169 | #[derive(Serialize, Deserialize, Debug, Clone)]
| ^^^^^^^^^^^ the trait std::convert::From<std::vec::Vec<u8>>
is not implemented for std::vec::Vec<std::vec::Vec<u8>>
|
= help: the following implementations were found:
<std::vec::Vec as std::convert::From<serde_bytes::ByteBuf>>
<std::vec::Vec as std::convert::From<std::collections::BinaryHeap>>
<std::vec::Vec as std::convert::From<std::boxed::Box<[T]>>>
<std::vec::Vec as std::convert::From<&'a str>>
and 6 others
= note: required by serde_bytes::deserialize
...
Hi,
i have a message to send in the form of:
type PubKey = Vec<u8>;
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag="kind")]
pub enum Envelope<T> {
Greeting {
#[serde(with="serde_bytes")]
id: PubKey,
#[serde(with="serde_bytes")]
shared: PubKey,
thin: bool
},
AllKnown {
all_known: Vec<PubKey>
},
}
As you can see, PubKey
is only a Vec<u8>
, what is the most idiomatic way to pack this. I do not want to put the Vec<u8>
in a struct to use serde_bytes
.
Is there something i am missing?
It would be really neat to annotated all_known
just with #[serde(with="serde_bytes")]
.
It seems a pretty common problem to be to have an array of byte arrays.
These traits are not intended to be implemented outside of this crate.
The README has this line:
Rust support for specialization is being tracked in rust-lang/rust#31844. Once it lands in the stable compiler Serde will be deprecating these wrapper types in favor of optimizing
&[u8]
andVec<u8>
out of the box.
Serialization in Go has taken a similar tack, always serialising []uint8
as []bytes
, but this has significant knock-on effects which can be quite painful.
As an example, serializing []int8
(&[i8]
) to JSON results in an array of numbers, while []uint8
(&[u8]
) results in a base64 blob (credit @edef1c for the example).
Taking this choice away from the user, by having specialization rather than explicit wrapper types, may make it difficult to impossible to comply with schemas that originate somewhere other than the Rust project using Serde - in Go, the solution is an expensive promotion to 16-bit integers, which costs a copy.
For example, I hava a strcut defined as follow:
pub struct VariableLength(u32);
And the field using a variable length encoding scheme, how can I implement Serializer/Deserializer for it?
VaribaleLength <---> Vec<u8>/&[u8]
Thanks a lot.
drop-in replacement solution is in https://crates.io/crates/serde_bytes_ng
I hope it can be merged into this, instead of maintaining another crate.
When I try to use:
#[derive(Serialize, Deserialize, Debug)]
struct Test<'a>
{
#[serde(with = "serde_bytes")]
a: &'a [u8]
}
I am getting compile errors:
135 | #[derive(Serialize, Deserialize, Debug)]
| ^^^^^^^^^^^ the trait `std::convert::From<std::vec::Vec<u8>>` is not implemented for `&[u8]`
|
= help: the following implementations were found:
<&'a [u8] as std::convert::From<serde_bytes::Bytes<'a>>>
= note: required by `serde_bytes::deserialize`
When I try:
#[derive(Serialize, Deserialize, Debug)]
struct Test<'a>
{
a: serde_bytes::Bytes<'a>
}
Then:
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'de` due to conflicting requirements
--> main.rs:135:21
|
135 | #[derive(Serialize, Deserialize, Debug)]
It would be more efficient to support deserialisation into &[u8] to avoid copies similarly as I can deserialise into &str. Or I am missing something?
This doesn't look critical, but it might be nice to fix the tests to do the right thing here, if for no purpose than to allow testing this configuration to ensure the crate works properly.
cargo +stable test --no-default-features
Updating crates.io index
Downloaded bincode v1.2.1
Compiling proc-macro2 v1.0.8
Compiling serde v1.0.104
Compiling unicode-xid v0.2.0
Compiling byteorder v1.3.4
Compiling syn v1.0.14
Compiling quote v1.0.2
Compiling serde_derive v1.0.104
Compiling bincode v1.2.1
Compiling serde_test v1.0.104
Compiling serde_bytes v0.11.3 (/home/kent/.cpanm/work/1582251421.24569/serde_bytes-0.11.3)
error[E0432]: unresolved import `serde_bytes::ByteBuf`
--> tests/test_partialeq.rs:1:26
|
1 | use serde_bytes::{Bytes, ByteBuf};
| ^^^^^^^ no `ByteBuf` in the root
error[E0432]: unresolved import `serde_bytes::ByteBuf`
--> tests/test_derive.rs:1:19
|
1 | use serde_bytes::{ByteBuf, Bytes};
| ^^^^^^^ no `ByteBuf` in the root
error[E0432]: unresolved import `serde_bytes::ByteBuf`
--> tests/test_serde.rs:1:19
|
1 | use serde_bytes::{ByteBuf, Bytes};
| ^^^^^^^ no `ByteBuf` in the root
error[E0425]: cannot find function `deserialize` in crate `serde_bytes`
--> tests/test_derive.rs:5:21
|
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
| ^^^^^^^^^^^ help: a function with a similar name exists: `serialize`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0432`.
error: aborting due to previous error
For more information about this error, try `rustc --explain E0432`.
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
--> tests/test_derive.rs:23:5
|
23 | cow_bytes: Cow<'a, Bytes>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
|
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
= note: only the last field of a struct may have a dynamically sized type
error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
--> tests/test_derive.rs:23:5
|
23 | cow_bytes: Cow<'a, Bytes>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
= note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
= note: required because it appears within the type `serde_bytes::bytes::Bytes`
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
= note: only the last field of a struct may have a dynamically sized type
error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
--> tests/test_derive.rs:5:21
|
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
| ^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
|
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
= note: only the last field of a struct may have a dynamically sized type
error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
--> tests/test_derive.rs:5:21
|
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
| ^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
= note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
= note: required because it appears within the type `serde_bytes::bytes::Bytes`
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
= note: only the last field of a struct may have a dynamically sized type
error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
--> tests/test_derive.rs:5:10
|
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
| ^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
|
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required by `std::borrow::Cow`
error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
--> tests/test_derive.rs:5:10
|
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
| ^^^^^^^^^ doesn't have a size known at compile-time
|
= help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
= note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
= note: required because it appears within the type `serde_bytes::bytes::Bytes`
= note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
= note: required by `std::borrow::Cow`
error: aborting due to 8 previous errors
Some errors have detailed explanations: E0277, E0425, E0432.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error: build failed
Currently Travis will only run cargo test
with default features.
It would be much appreciated.
While this crate helps serialize bytes in binary formats efficiently, human-readable formats will still have them as vectors of integers. Is efficient serialization in human-readable formats something potentially fitting for this crate, or should it be implemented elsewhere?
That may include:
Cow
is handled in this way in serde
. But serde-bytes
deserializes Cow
by borrowing. As a result, this compiles:
#[derive(Serialize, Deserialize)]
struct A<'a> {
a: Cow<'a, [u8]>,
}
But this does not compile:
#[derive(Serialize, Deserialize)]
struct A<'a> {
#[serde(with = "serde_bytes")]
a: Cow<'a, [u8]>,
}
The error message generated by compiler:
error: lifetime may not live long enough
--> src/main.rs:4:21
|
4 | #[derive(Serialize, Deserialize)]
| ^^^^^^^^^^^
| |
| lifetime `'de` defined here
| associated function was supposed to return data with lifetime `'a` but it is returning data with lifetime `'de`
5 | struct A<'a> {
| -- lifetime `'a` defined here
|
= help: consider adding the following bound: `'de: 'a`
= note: this error originates in the derive macro `Deserialize` (in Nightly builds, run with -Z macro-backtrace for more info)
Isn't it more reasonable to handle Cow
in a way similar to serde
so that deserializing Cow
does not borrow?
Per common convention, Bytes
should implement ToOwned
that converts them to ByteBuf
.
In particular, this would allow easily using Cow<Bytes>
in contexts where you want either borrowed or owned data.
…or perhaps a SmallByteBuf<N>
wrapper?
Now that const_generics
are stable, I wonder if it is possible to implement support for [u8; N]
arrays (serde
seems to support them)?
I'm relatively new to Rust so bear with me. I'm working on a project that provides a Python API to Rust code via Pyo3. In my project I'm trying to enable fast serialization and deserialization using bincode
. I read this crate can mitigate the performance issues I'm seeing (bincode
takes 3 times longer than serialization/deserialization with serde_json
).
It's not clear to me how to take advantage of this crate when I don't use a struct field to hold the data.
For reference, this is what the unedited, actual serialization and deserialization code snippet looks like, with the Pyo3 API intact:
pub fn to_bincode(&self) -> PyResult<Vec<u8>> {
Ok(serialize(&self).unwrap())
}
#[classmethod]
pub fn from_bincode(_cls: &PyType, encoded: Vec<u8>) -> PyResult<Self> {
Ok(deserialize(&encoded).unwrap())
}
With the API removed, I suppose would be the analogous code:
pub fn to_bincode(&self) -> Vec<u8> {
serialize(&self).unwrap()
}
pub fn from_bincode(encoded: Vec<u8>) -> Self {
deserialize(&encoded).unwrap()
}
See also: PyO3/pyo3#100
How can I utilize this crate in this case?
Some of the conversions are hard to discover from rustdoc.
let v: Vec<u8> = bytebuf.into();
This would be more discoverable as:
let v = bytebuf.into_vec();
serde supports Arc with the 'rc' feature. Is it possible to have support for Arc<Vec< u8>>?
They could coexists with ByteBuf::from and ByteBuf::into_vec if you think these functions are clearer in documentation. I never had problems with looking for From trait implementations.
Now is not possible to use it with generic functions which uses things like T: From<Vec<u8>>
. There is reason why there are traits like From trait in rust.
And solution to problems like this #14 require even more extra glue code as before.
That would be useful while writing tests, to compare a Vec<u8>
or a bytestring literal to a Bytes
or a ByteBuf
without trying to wrap things up.
Hi!
I stumbled around some confusing behaviour with regards to no_std
builds:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
struct Testing<'a> {
#[serde(with = "serde_bytes")]
testing: &'a [u8],
}
Compiles if the std
or alloc
feature is enabled, but doesn't compile if it isn't.
I do not understand why the deserialize
function is behind such a feature flag.
I understand that serde_bytes
only changes the serialization of &[u8]
since serde's implementation is already optimized, so you can get around the issue with
#[derive(Deserialize, Serialize)]
struct Testing<'a> {
#[serde(serialize_with = "serde_bytes::serialize")]
testing: &'a [u8],
}
But:
#[serde(with = "serde_bytes")]
rustc
can't understand the problem (though recent version do correctly point out the feature flag).Can this feature-gate be removed?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.