atilaneves / cerealed Goto Github PK

serde is serialization framework for Rust.

https://github.com/serde-rs/serde

Is it possible to serialize private fields of classes and structs ?

in order for dub to recognise the release the "v" must be lowercase.

I didn't want to go through the trouble of making a full pull request for so little but on the other hand it's important that example code work without any problem:

diff --git a/README.md b/README.md
index 192b10f..a700ea0 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ Example usage:
     cerealiser ~= cast(ubyte)42;
     assert(cerealiser.bytes == [ 0, 0, 0, 5, 42]);

-    auto deceralizer = Decerealizer([ 0, 0, 0, 5, 42]); //US spelling works too
+    auto decerealizer = Decerealizer([ 0, 0, 0, 5, 42]); //US spelling works too
     assert(decerealizer.value!int == 5);
     assert(decerealizer.value!ubyte == 42);

with DMD64 D Compiler v2.075.0

struct Sample {}
void main(string[] args) {
  import cerealed;
  import std.file : read;
  const(ubyte)[] bytes = cast(const(ubyte)[])(read("samples.dat"));
  bytes.decerealise!(Sample[]);
}

../../.dub/packages/cerealed-0.6.8/cerealed/src/cerealed/cereal.d(192,18): Error: variable length used before set

Did I miss something?

Some protocols/file formats may use a different endianness than the one native to the platform it is (de)serialised on.

Should a Big-Endian architecture ever gain D support, an application serialising data on it and then deserialising it on a Little-Endian platform will fail to do so.

File formats, such as PNG, use Big Endian ("network byte ordering"), yet the dominant Endianness today is Little Endian. Having a shortcut for saying "Please bswap if I need it" would be handy, instead of having to define this as custom behaviour.

Issue #16 is still present in dub package.

Thanks for the great lib, using it everyday now.

One of my associative arrays went above 65535 elements and I got an overflow error from cerealise, there's code below to reproduce the issue. I noticed the ushorts in cerealed's source code and tried somewhat haphazardly to replace them with size_t, got it half working but not fully.

Would it be thinkable to have an optional type parameter, default ushort (as now) ? Or any other solution supporting bigger assoc. arrays ?
.
overflow use case on assoc. arrays:

import cerealed;

import std.algorithm;
import std.range;
import std.stdio;

void main()
{
  test( 10 );  // ok
  test( 65535 );  // ok
  test( 65536 );  // overflow
}

ubyte[] test( in size_t n )
{
  bool[size_t] data;
  n.iota.each!( (i) { data[ i ] = true; } );

  writeln( "About to cerealise data for n: ", data.length );
  auto ret = cerealise( data );
  writeln( "Done." );
  
  return ret;
}

The following Testcase fails at runtime:

struct TestStruct {
    ubyte len;
    @ArrayLength("len") char[] foo;
}

auto decerealiser = Decerealiser([2, 1, 2]);
// CerealException: "@ArrayLength of 2 units of type dchar (4 bytes) larger than remaining byte array (2 bytes)"
auto ts = decerealiser.value!TestStruct;
assert(ts.foo == ['\x01', '\x02']);
assert(ts.foo.length == 2);

module test;

import cerealed;
import std.stdio;

struct Pair {
  string s1;
  int a;
}


void main() {
  auto p = Pair("foo", 5);

  int[Pair] map;
  map[p] = 105;

  auto ser = new Cerealiser();
  ser ~= map;

  auto deser = new Decerealiser(ser.bytes);
  int[Pair] outcu = deser.value!(int[Pair]);
  writeln(outcu);
}

Expected output: [Pair("foo", 5): 105]
Actual output : [Pair("", 0): 105]

This broken down testcase doesn't compile, as long there is a alias this.

    struct TestStruct {
        char[] txt;
        alias txt this;
    }

    auto decerealiser = Decerealiser([0, 2, 0x90, 0x91]);
    auto ts = decerealiser.value!TestStruct;
    assert(ts.txt == ['\x90', '\x91']);
    assert(ts.txt.length == 2);

Why does Cerealed limit max size of serialized data to 64KiB?

Consider:

import std.stdio;

import cerealed;

struct ubyted {
    ubyte[] payload;
}
void main() {
    ubyte[] buff = new ubyte[2 ^^ 16];
    Cerealiser cer;

    cer ~= buff;
}

This code will fail with assert error on line 87 in cereal.d
Error: cerealed-0.6.6/src/cerealed/cereal.d(87): overflow

A buffer with length less with one will not trigger assert.
Is it by design, or is it a bug?

Here is the test case:

import cerealed;

struct Foo { string value; }

auto data = Foo("bar").cerealise;
auto f1 = data.decerealise!Foo;
assert(f1.value == "bar");
//f1.value = f1.value.idup;

auto data2 = Foo("baz").cerealise;
data[] = data2[];
auto f2 = data.decerealise!Foo;
assert(f2.value == "baz");
//f2.value = f2.value.idup;
assert(f1.value == "bar");

When string is deserialized it points to the original muttable buffer so when other data come to it, string is modified even though it's immutable.

Attempting to serialize a ubyte[] array results in a compiler error. Line 130 in cereal.d attempts to call dup on length, which isn't an array.

Test Program:

void main() {
    auto test = new ubyte[5];


    auto cerealiser = Cerealiser();
    cerealiser ~= test;

    auto deceralizer = Decerealizer(cerealiser.bytes);
    auto testcpy = deceralizer.value!(ubyte[]);

    assert(test == testcpy);
}

Output

> dub build
Target cerealed 0.6.1 is up to date. Use --force to rebuild.
Building test ~master configuration "application", build type debug.
Compiling using dmd...
../.dub/packages/cerealed-0.6.1/src/cerealed/cereal.d(130): Error: template object.dup cannot deduce function from argument types !()(short), candidates are:
/usr/include/dmd/druntime/import/object.d(1785):        object.dup(T : V[K], K, V)(T aa)
/usr/include/dmd/druntime/import/object.d(1821):        object.dup(T : V[K], K, V)(T* aa)
/usr/include/dmd/druntime/import/object.d(3122):        object.dup(T)(T[] a) if (!is(const(T) : T))
/usr/include/dmd/druntime/import/object.d(3138):        object.dup(T)(const(T)[] a) if (is(const(T) : T))
/usr/include/dmd/druntime/import/object.d(3149):        object.dup(T : void)(const(T)[] a)
../.dub/packages/cerealed-0.6.1/src/cerealed/cereal.d(110): Error: template instance cerealed.cereal.decerealiseArrayImpl!(Decerealiser, ubyte[], short) error instantiating
../.dub/packages/cerealed-0.6.1/src/cerealed/decerealiser.d(63):        instantiated from here: grain!(Decerealiser, ubyte[], short)
source/app.d(18):        instantiated from here: value!(ubyte[], short)
FAIL .dub/build/application-debug-linux.posix-x86_64-dmd_2068-52E1E9557A38487A968910C60DCC3586/ test executable
Error executing command build:
dmd failed with exit code 1.

I see that there is a sample for serializing through base class reference:

Cereal.registerChildClass!ChildClass;
BaseClass obj = ChildClass(3, 7);
assert(obj.cerealise == [0, 0, 0, 3, 0, 0, 0, 7]);

However, as the assert shows, the derived class name isn't stored anywhere. So when deserializing, you need to know the type anyway. Imagine serializing a BaseClass[]. Some elements of the array are instances of BaseClass, some are instances of ChildClass. Shouldn't in the case of classes an identifier be stored saying what is the real class that is being stored? Something like "assert(obj.cerealise == ["BaseClass", 0, 0, 0, 3, 0, 0, 0, 7]);", so that later on in the code you can do:

Cereal.registerChildClass!ChildClass;
BaseClass obj = datastream.decerealise()

and obj will be actually the ChildClass instance?

Cerealed seems to have no support for different byte orders.
The library should clearly support both little and big endian.

Hello Atila,
As you suggested I've signed in and posted my feature request here.
I would like to use cerealed with a struct like this:

import cerealed;

struct nested_associative_array { nested_associative_array[int] x; }

struct some_struct {
    string[] x;
    int[][] y;
    nested_associative_array[] z;
};

void main() {
    some_struct original, restored;

    auto enc = Cerealiser();
    enc ~= original;
    auto dec = Decerealiser(enc.bytes);
    restored = dec.value!(some_struct);

    assert(original==restored);
}

The serialization process compiles fine, but the restoring part fails to compile.
I've tried to figure out how to do it, but I am still a beginner on D.
How difficult do you think it is?
The alternative, orange, works fine, but the struct I'm trying to export to disk is a few hundreds of MB in size and orange takes hours to complete and may also run out of memory. Your implementation, instead, is really fast and efficient!