Code Monkey home page Code Monkey logo

spanjson's Introduction

SpanJson

NuGet

See https://github.com/Tornhoof/SpanJson/wiki/Performance for Benchmarks

What is supported

  • Serialization and Deserialization into/from byte arrays, strings, TextWriter/TextReader and streams
  • Serialization and Deserialization of Arrays, Lists, Complex types of the following Base Class Library types:

sbyte, Int16, Int32, Int64, byte, UInt16, UInt32, UInt64, Single, Double, decimal, bool, char, DateTime, DateTimeOffset, TimeSpan, Guid, string, Version, Uri, Tuple<,>,ValueTuple<,>, KeyValuePair<,>

  • Public Properties and Fields are considered for serialization/deserialization

  • DateTime{Offset} is in ISO8601 mode with profile https://www.w3.org/TR/NOTE-datetime

  • Dynamics

  • Enums (string and integer, for integer see section Custom Resolver), incl. Flags

  • Anonymous types

  • Dictionary, ConcurrentDictionary with string/int/enum as key, the enum is formatted as a string.

  • Serialization/Deserialization of most IEnumerable types (Stack and ConcurrentStack are not supported)

  • Support for [DataMember(Name="MemberName")] to set field name

  • Support for [IgnoreDataMember] to ignore a specific member

  • Support for ShouldSerializeXXX pattern to decide at runtime if a member should be serialized

  • Support for [EnumMember] to specify the string value of the enum value

  • Support for Immutable Collections, full Serialization/Deserialization for ImmutableList, ImmutableArray, ImmutableDictionary, ImmutableSortedDictionary. ImmutableStack is not supported.

  • Support for read-only collections, ReadOnlyCollection, ReadOnlyDictionary, they are deserialized into a writeable type (i.e. List or Dictionary), then the read-only version is created via an appropriate constructor overload.

  • Support for tuples currently excludes the last type with 8 arguments (TRest)

  • Support for annotating a constructor with [JsonConstructor] to use that one instead of assigning members during deserialization

  • Support for custom serializers with [JsonCustomSerializer] to use that one instead of the normal formatter, see examples below

  • Support for Base64 encoded byte arrays, see the Custom Resolvers example below

  • Support for annotating a IDictionary<string,object> with [JsonExtensionData]. Serialization will write all values from the dictionary as additional attributes. Deserialization will deserialize all unknown attributes into it. This does not work together with the Dynamic Language Runtime (DLR) support or the [JsonConstructor] attribute. See Example below. The Dictionary will also honor the Case Setting (i.e. CamelCase) and null behaviour for the dictionary keys.

  • Pretty printing JSON

  • Minify JSON

  • Different 'Resolvers' to control general behaviour:

    • Exclude Nulls with Camel Case: ExcludeNullsCamelCaseResolver
    • Exclude Nulls with Original Case (default): ExcludeNullsOriginalCaseResolver
    • Include Nulls with Camel Case: IncludeNullsCamelCaseResolver
    • Include Nulls with Original Case: IncludeNullsOriginalCaseResolver
  • Custom Resolvers to control behaviour much more detailed.

How to use it

Synchronous API:

var result = JsonSerializer.Generic.Utf16.Serialize(input);
var result = JsonSerializer.NonGeneric.Utf16.Serialize(input);
var result = JsonSerializer.Generic.Utf16.Deserialize<Input>(input);
var result = JsonSerializer.NonGeneric.Utf16.Deserialize(input, typeof(Input));

var result = JsonSerializer.Generic.Utf8.Serialize(input);
var result = JsonSerializer.NonGeneric.Utf8.Serialize(input);
var result = JsonSerializer.Generic.Utf8.Deserialize<Input>(input);
var result = JsonSerializer.NonGeneric.Utf8.Deserialize(input, typeof(Input));

// The following methods return an ArraySegment from the ArrayPool, you NEED to return it yourself after working with it.
var result = JsonSerializer.Generic.Utf16.SerializeToArrayPool(input);
var result = JsonSerializer.NonGeneric.Utf16.SerializeToArrayPool(input);
var result = JsonSerializer.Generic.Utf8.SerializeToArrayPool(input);
var result = JsonSerializer.NonGeneric.Utf8.SerializeToArrayPool(input);

Asynchronous API:

ValueTask result = JsonSerializer.Generic.Utf16.SerializeAsync(input, textWriter, cancellationToken);
ValueTask result = JsonSerializer.NonGeneric.Utf16.SerializeAsync(input, textWriter, cancellationToken);
ValueTask<Input> result = JsonSerializer.Generic.Utf16.DeserializeAsync<Input>(textReader,cancellationToken);
ValueTask<object> result = JsonSerializer.NonGeneric.Utf16.DeserializeAsync(textReader,typeof(Input),cancellationToken);
ValueTask result = JsonSerializer.Generic.Utf8.SerializeAsync(input, stream, cancellationToken);
ValueTask result = JsonSerializer.NonGeneric.Utf8.SerializeAsync(input, stream, cancellationToken);
ValueTask<Input> result = JsonSerializer.Generic.Utf8.DeserializeAsync<Input>(input, stream, cancellationToken);
ValueTask<object> result = JsonSerializer.NonGeneric.Utf8.DeserializeAsync(input, stream, typeof(Input) cancellationToken);

To use other resolvers use the appropriate overloads,e.g.:

var serialized = JsonSerializer.NonGeneric.Utf16.Serialize<Input, IncludeNullsOriginalCaseResolver<char>>(includeNull);

Pretty Printing:

var pretty = JsonSerializer.PrettyPrinter.Print(serialized); // this works by reading the JSON and writing it out again with spaces and line breaks

Minify:
var minified = JsonSerializer.Minifier.Minify(serialized); // this works by reading the JSON and writing it out again without spaces and line breaks

Full example:

using System;
using SpanJson;

namespace Test
{
    public class Program
    {
        private static void Main(string[] args)
        {

            var input = new Input { Text = "Hello World" };

            var serialized = JsonSerializer.Generic.Utf16.Serialize(input);

            var deserialized = JsonSerializer.Generic.Utf16.Deserialize<Input>(serialized);
        }
    }

    public class Input
    {
        public string Text { get; set; }
    }
}
using System;
using SpanJson;

namespace Test
{
    // This JsonConstructorAttribute assumes that the constructor parameter names are the same as the member names (case insensitive comparison, order is not important)
    public class DefaultDO
    {
        [JsonConstructor]
        public DefaultDO(string key, int value)
        {
            Key = key;
            Value = value;
        }

        public string Key { get; }
        public int Value { get; }
    }

    // This JsonConstructorAttribute allows overwriting the matching names of the constructor parameter names to allow for different member names vs. constructor parameter names, order is important here
    public readonly struct NamedDO
    {
        [JsonConstructor(nameof(Key), nameof(Value))]
        public NamedDO(string first, int second)
        {
            Key = first;
            Value = second;
        }


        public string Key { get; }
        public int Value { get; }
    }
}
// Type with a custom serializer to (de)serialize the long value into/from string
public class TestDTO
{
    [JsonCustomSerializer(typeof(LongAsStringFormatter), "Hello World")]
    public long Value { get; set; }
}

// Serializes the Long into a string
public sealed class LongAsStringFormatter : ICustomJsonFormatter<long>
{
    public static readonly LongAsStringFormatter Default = new LongAsStringFormatter();
    
    public object Arguments {get;set;} // the Argument from the attribute will be assigned

    public void Serialize(ref JsonWriter<char> writer, long value)
    {
        StringUtf16Formatter.Default.Serialize(ref writer, value.ToString(CultureInfo.InvariantCulture));
    }

    public long Deserialize(ref JsonReader<char> reader)
    {
        var value = StringUtf16Formatter.Default.Deserialize(ref reader);
        if (long.TryParse(value, out long longValue))
        {
            return longValue;
        }

        throw new InvalidOperationException("Invalid value.");
    }

    public void Serialize(ref JsonWriter<byte> writer, long value)
    {
        StringUtf8Formatter.Default.Serialize(ref writer, value.ToString(CultureInfo.InvariantCulture));
    }

    public long Deserialize(ref JsonReader<byte> reader)
    {
        var value = StringUtf8Formatter.Default.Deserialize(ref reader);
        if (long.TryParse(value, out long longValue))
        {
            return longValue;
        }

        throw new InvalidOperationException("Invalid value.");
    }
}
// It's possible to annotate custom types a custom formatter to always use the custom formatter 
[JsonCustomSerializer(typeof(TwcsCustomSerializer))]
public class TypeWithCustomSerializer : IEquatable<TypeWithCustomSerializer>
{
    public long Value { get; set; }
}

// Instead of copying the implementation of for serialize/deserialize for utf8/utf16
// it is possible to use the writer/reader methods which support both, there is no or only a very minor performance difference
public sealed class TwcsCustomSerializer : ICustomJsonFormatter<TypeWithCustomSerializer>
{
    public static readonly TwcsCustomSerializer Default = new TwcsCustomSerializer();

    public object Arguments { get; set; }

    private void SerializeInternal<TSymbol>(ref JsonWriter<TSymbol> writer, TypeWithCustomSerializer value) where TSymbol : struct
    {
        if (value == null)
        {
            writer.WriteNull();
            return;
        }

        writer.WriteBeginObject();

        writer.WriteName(nameof(TypeWithCustomSerializer.Value));

        writer.WriteInt64(value.Value);

        writer.WriteEndObject();
    }

    public void Serialize(ref JsonWriter<byte> writer, TypeWithCustomSerializer value)
    {
        SerializeInternal(ref writer, value);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    private TypeWithCustomSerializer DeserializeInternal<TSymbol>(ref JsonReader<TSymbol> reader) where TSymbol : struct
    {
        if (reader.ReadIsNull())
        {
            return null;
        }

        reader.ReadBeginObjectOrThrow();
        var result = new TypeWithCustomSerializer {Value = reader.ReadInt64()};
        reader.ReadEndObjectOrThrow();
        return result;
    }

    public TypeWithCustomSerializer Deserialize(ref JsonReader<byte> reader)
    {
        return DeserializeInternal(ref reader);
    }

    public void Serialize(ref JsonWriter<char> writer, TypeWithCustomSerializer value)
    {
        SerializeInternal(ref writer, value);
    }

    public TypeWithCustomSerializer Deserialize(ref JsonReader<char> reader)
    {
        return DeserializeInternal(ref reader);
    }
}
// Below class will serialize Key and Value and any additional key-value-pair from the dictionary
public class ExtensionTest
{
    public string Key;
    public string Value;

    [JsonExtensionData]
    public IDictionary<string, object> AdditionalValues { get; set; }
}

ASP.NET Core 6.0+ Formatter

You can enable SpanJson as the default JSON formatter in ASP.NET Core 6.0+ by using the Nuget package SpanJson.AspNetCore.Formatter. To enable it, add one of the following extension methods to the AddMvc() call in ConfigureServices

  • AddSpanJson for a resolver with ASP.NET Core 6.0 defaults: IncludeNull, CamelCase, Integer Enums
  • AddSpanJsonCustom for a custom resolver (one of the default resolvers or custom)
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc().AddSpanJson();
}

AddSpanJson is the closest in behaviour compared to the default JSON.NET formatter, this used the AspNetCoreDefaultResolver type. Note: This clears the Formatter list, if you have other formatters, e.g. JSON Patch or XML, you need to re-add them.

Custom Resolver

As each option is a concrete class it is infeasible to supply concrete classes for each possible option combination. To support a custom combination implement your own custom formatter resolvers

public sealed class CustomResolver<TSymbol> : ResolverBase<TSymbol, CustomResolver<TSymbol>> where TSymbol : struct
{
    public CustomResolver() : base(new SpanJsonOptions
    {
        NullOption = NullOptions.ExcludeNulls,
        NamingConvention = NamingConventions.CamelCase,
        EnumOption = EnumOptions.Integer,
        ByteArrayOptions = ByteArrayOptions.Base64
    })
    {
    }
}

and pass this type just the same as e.g. ExcludeNullsCamelCaseResolver

TODO

  • Improve async deserialization/serialization: Find a way to do it streaming instead of buffering.

spanjson's People

Contributors

havunen avatar olsondev avatar reris avatar tornhoof avatar zlatanov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spanjson's Issues

Unable to deserialize from Gzip Stream

Hi @Tornhoof ,

When I tried to de-serialize from Gzip decompressed stream, I got this exception:
SpanJson.JsonParserException: Error Reading JSON data: 'ExpectedBeginObject' at position: '0'.

Could you please help to check?
Below is stack trace and the unit test.

SpanJson.JsonParserException: Error Reading JSON data: 'ExpectedBeginObject' at position: '0'.
    at SpanJson.JsonReader`1.ThrowJsonParserException(ParserError error) in C:\projects\spanjson\SpanJson\JsonReader.cs:line 50
   at lambda_method(Closure , JsonReader`1& )
   at SpanJson.Formatters.ComplexClassFormatter`3.Deserialize(JsonReader`1& reader) in C:\projects\spanjson\SpanJson\Formatters\ComplexClassFormatter.cs:line 21
   at SpanJson.JsonSerializer.Generic.Inner`3.InnerDeserialize(ReadOnlySpan`1& input) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 124
   at SpanJson.JsonSerializer.Generic.Inner`3.InnerDeserialize(Memory`1 memory) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 169
   at SpanJson.JsonSerializer.Generic.Inner`3.InnerDeserializeAsync(Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 142
   at SpanJson.JsonSerializer.Generic.Utf8.DeserializeAsync[T,TResolver](Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 543
   at SpanJson.JsonSerializer.Generic.Utf8.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 514
   at JsonSerializationTests.GzipStreamTests.Deserialize_From_GzipStream_Vs_Newtonsoft_Json() 
using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Text;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;

namespace JsonSerializationTests
{
    [TestClass]
    public class GzipStreamTests
    {
        [TestMethod]
        public async Task Serialize_To_GzipStream_And_Deserialize_From_GzipStream()
        {
            using (var stream = new MemoryStream())
            using (var compressedStream = new GZipStream(stream, CompressionLevel.Fastest, true))
            {
                await SpanJson.JsonSerializer.Generic.Utf8.SerializeAsync(new Msg {Body = "HelloWorld"}, compressedStream);

                stream.Seek(0, SeekOrigin.Begin);
                using(var decompressStream = new GZipStream(stream,CompressionMode.Decompress, true))
                {
                    var msg = await SpanJson.JsonSerializer.Generic.Utf8.DeserializeAsync<Msg>(decompressStream);
                    Assert.IsNotNull(msg?.Body);
                }
            }
        }

        [TestMethod]
        public async Task Deserialize_From_GzipStream_Vs_Newtonsoft_Json()
        {
            var x = new Msg {Body = "HelloWorld"};
            var jsonSerializer = new JsonSerializer();

            using (var stream = new MemoryStream())
            using (var compressedStream = new GZipStream(stream, CompressionLevel.Fastest, true))
            {
                using (var textWriter = new StreamWriter(compressedStream, Encoding.UTF8, 1024, true))
                {
                    jsonSerializer.Serialize(textWriter, x);
                }

                //test with newtonsoft.json
                stream.Seek(0, SeekOrigin.Begin);
                using(var decompressStream = new GZipStream(stream,CompressionMode.Decompress, true))
                {
                    using(var textReader = new StreamReader(decompressStream, Encoding.UTF8, false, 1024, true))
                    using (var reader = new JsonTextReader(textReader))
                    {
                        var msg = jsonSerializer.Deserialize<Msg>(reader);
                        Assert.IsNotNull(msg?.Body);
                    }
                }

                //test with SpanJson
                stream.Seek(0, SeekOrigin.Begin);
                using(var decompressStream = new GZipStream(stream,CompressionMode.Decompress, true))
                {
                    //The below line got Exception
                    var msg = await SpanJson.JsonSerializer.Generic.Utf8.DeserializeAsync<Msg>(decompressStream); 

                    Assert.IsNotNull(msg?.Body);
                }
            }
        }
    }

    public class Msg
    {
        public string Body { get; set; }
    }
}

Deserialization perf for partial response objects

Hi,
I was wondering if there is anything that could speed up deserialization when my deserialized object contains only some of the properties of json object or when JSON object has an array of objects and I'm only interested in one out of 10 properties of objects in such array?

Incorrect serialization of surrogate pairs when writing utf8.

The current implementation handles values in ucs2, but cannot handle surrogate pairs.
The code is currently is encoding each pair as a separate code point and as a result is not handling correctly.
Thus "๐Ÿ’ฉ" will serialize as the following "๏ฟฝ๏ฟฝ".

You can detect surrogate pair with the following logic::

(uint)c - 0xD800u < 0x0800u

(changed the constant, the range for surrogate its 0xD800-0xDFFF).
I think it is safe to assume that a high surrogate pair character always is followed by a low surrogate pair character, so you can probably avoid testing both as malformed utf16, will only encode to malformed utf8.

Incorrect serialization behavior for long.MinValue

The WriteUtf*Int64Internal internal method special case for long.MinValue, but they do not return early. and so it will write the long.MinValue and then it will -long.MinValue.
So this object:

        public long Long {get;set;} = long.MinValue;
}

will serialize to the following json {"Long":-92233720368547758089223372036854775808}
You can see the bug here::
https://github.com/Tornhoof/SpanJson/blob/master/SpanJson/JsonWriter.Utf16.cs#L51

The special case is actually unnecessary and you can simplify the code. Even though negating it does nothing, casting it to ulong yields the correct value.

        public void WriteUtf16(long value)
        {
            ref var pos = ref _pos;
            if (value < 0)
            {
                if (pos > _chars.Length - 1)
                {
                    Grow(1);
                }

                _chars[pos++] = '-';
                value = unchecked(-value);
            }

            WriteUtf16((ulong)value);
      }

Improve Array Sizing estimation

Background
Currently SpanJson saves both the last serialization/deserialization size of an object to a static variable and will reuse that size for following serialization/deserialization runs to set the initial buffer size.
This is based on the (tested) assumption that many serialized complex types have a similar size for a specific formatter.

In combination with the fact, that the ArrayPool returns values in an power of 2, e.g. size is 27, arraypool returns 32, we have a fairly good initial bet so we keep the resizing to a minimum.

Problem
Unfortunately this algorithm does not take into account, that during string writing of utf8 we highly overestimate the size of the string, as we preallocate the array size for the worst case, e.g. all characters are full blown supplementary plane chars (e.g. 4 bytes). This situation is highly unlikely, as most strings will not be emoticons only ;)

Solution
Instead of using the actual size of the written/read output, take the the last resized array size.
This should improve the UTF8 serialization/deserialization performance for several edge cases.

Serialize Enum as int

Currently the behavior of serializing Enum is to convert it to string. However I think Enums should be serialized to int.

JsonParserException: 'Error Reading JSON data: 'ExpectedDoubleQuote' at position: X

This works OK:

JsonSerializer.Generic.Utf8.Deserialize<Dictionary<string, object>>(System.Text.Encoding.UTF8.GetBytes(@"{""a"": 1,""b"": ""2""}"));

But this fails:

JsonSerializer.Generic.Utf8.Deserialize<Dictionary<string, object>>(System.Text.Encoding.UTF8.GetBytes(@"{""a"": 1, ""b"": ""2""}"));

In case i add space after "1," the above exception is thrown.

Expected behavior:
insignificant spaces should be skipped without an error

The type initializer for 'Inner`3' threw an exception

The type initializer for 'Inner`3' threw an exception.@   at lambda_method(Closure , Object , Stream , CancellationToken )
   at SpanJson.JsonSerializer.NonGeneric.Inner`2.InnerSerializeAsync(Object input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.NonGeneric.cs:line 120
   at BeetleX.FastHttpApi.JsonResult.Write(PipeStream stream, HttpResponse response)
   at BeetleX.FastHttpApi.HttpResponse.OnWrite(PipeStream stream)
   at BeetleX.FastHttpApi.HttpResponse.BeetleX.FastHttpApi.IDataResponse.Write(PipeStream stream)

customer submitted a question and could not get more detailed information

Not compatible with netstandard2.0

I get the following error when trying to add this package to my project:

NU1202	Package SpanJson 2.0.4 is not compatible with netstandard2.0 (.NETStandard,Version=v2.0). Package SpanJson 2.0.4 supports: netcoreapp2.1 (.NETCoreApp,Version=v2.1)	

Is there any reason why this lib targets netcoreapp2.1 and cannot target netstandard2.0?

Plans for 2.0

Breaking Changes

  • Strong Naming is a breaking change.
  • removing NestingLimit from the API and using a property in the writer is a breaking change.

Features

  • Evaluate if #61 is possible.

[Question] JSON Constructor - can it be private?

I am wondering whether a JSONConstructor attribute will work with a private constructor?

The use case is that I have some business objects, that have non default constructors. In the ordinary case, when constructing the business object, these are the public constructors that should be used.

However if I want to allow the serialiser to be able to construct the business object during deserialisation, the fact that it doesn't have a public default constructor is problematic - meaning I must add a constructor to support the serialiser, and annotate it with the JSONConstructor attribute. This would work fine, but I would prefer not to expose this constructor in the public API as it shouldn't be used in the ordinary case of working with the business object.

Improve ComplexClass/StructFormatter

Currently the ComplexClass/StructFormatter uses a nested if/else on individual characters/bytes of the json attribute name to select the appropriate property to assign a value to.
While this is several times faster than a simple comparison with the full attribute name each time it's still not fast.
Current main branch for Answer model:

|                                      Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |
|-------------------------------------------- |---------:|----------:|----------:|-------:|----------:|
|       SerializeAnswerWithSpanJsonSerializer | 3.511 us | 0.0091 us | 0.0086 us | 0.8774 |   3.61 KB |
|   SerializeAnswerWithSpanJsonSerializerUtf8 | 4.264 us | 0.0062 us | 0.0058 us | 0.4501 |   1.87 KB |
|     DeserializeAnswerWithSpanJsonSerializer | 6.337 us | 0.0311 us | 0.0291 us | 0.4807 |      2 KB |
| DeserializeAnswerWithSpanJsonSerializerUtf8 | 7.530 us | 0.0035 us | 0.0031 us | 0.4807 |      2 KB |

On the codegen branch I wrote an extremely simple codegen (string.format) to create explicit (non-expressiontree) source files for each model. The deserialization logic then uses optimized if/else not on individual characters anymore but ulongs/uint etc. to simply read more characters/bytes and compare against constant integer values at the same time, e.g. for utf8 it's possible to compare 8 chars in a name span directly with an ulong.
This basically looks like this:

if (length == 15 && ReadUInt64(ref c, 0) == 8390054783061815140UL && ReadUInt32(ref c, 8) == 1868783461U && ReadUInt16(ref c, 12) == 28277 &&
ReadByte(ref c, 14) == 116)
{
result.down_vote_count = NullableInt32Utf8Formatter<ExcludeNullsOriginalCaseResolver<byte>>.Default.Deserialize(ref reader);
continue;
}

This method is ~20% faster than the original method:

|                                      Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |
|-------------------------------------------- |---------:|----------:|----------:|-------:|----------:|
|       SerializeAnswerWithSpanJsonSerializer | 3.641 us | 0.0077 us | 0.0065 us | 0.8774 |    3.6 KB |
|   SerializeAnswerWithSpanJsonSerializerUtf8 | 4.017 us | 0.0517 us | 0.0432 us | 0.4501 |   1.87 KB |
|     DeserializeAnswerWithSpanJsonSerializer | 4.892 us | 0.0198 us | 0.0176 us | 0.4807 |      2 KB |
| DeserializeAnswerWithSpanJsonSerializerUtf8 | 5.984 us | 0.0329 us | 0.0308 us | 0.4807 |      2 KB |

As the benchmark shows the difference in serialization is pretty much non-existant, but for deserialization it's obvious. This is actually the biggest increase in performance I've seen in a long time in SpanJson ;)
Unfortunately it's quite complicated to write that optimized code with expression trees, as it's not possible to assign ref values to variables in expression trees.
There are three solutions I can think of:

  1. Somehow mix expression trees and normal code to get around the above limitations, but because delegates can't be inlined this might actually be slow.
  2. Use roslyn to generate types at runtime like my current codegen, unfortunately it's not really possible these types on demand as roslyn would generate a unique dll in memory each time a new type is found.
  3. Use IL emit to generate them, this is probably the best solution but by far the hardest, as working with IL in .net core is kinda bad, e.g. you can't save your generated assemblies to disk, this makes the whole process several times slower as in .net fx

System.ArgumentException: The output byte buffer is too small to contain the encoded data

System.ArgumentException: The output byte buffer is too small to contain the encoded data, encoding 'Unicode (UTF-8)' fallback 'System.Text.EncoderReplacementFallback'.
Parameter name: bytes
   at System.Text.Encoding.ThrowBytesOverflow()
   at System.Text.Encoding.ThrowBytesOverflow(EncoderNLS encoder, Boolean nothingEncoded)
   at System.Text.UTF8Encoding.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
   at System.Text.UTF8Encoding.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount)
   at System.Text.Encoding.GetBytes(ReadOnlySpan`1 chars, Span`1 bytes)
   at SpanJson.JsonWriter`1.WriteUtf8Name(ReadOnlySpan`1& value) in C:\projects\spanjson\SpanJson\JsonWriter.Utf8.cs:line 473
   at SpanJson.Formatters.DictionaryFormatter`4.Serialize(JsonWriter`1& writer, TDictionary value, Int32 nestingLimit) in C:\projects\spanjson\SpanJson\Formatters\DictionaryFormatter.cs:line 40
   at SpanJson.Formatters.BaseFormatter.SerializeRuntimeDecisionInternal[T,TSymbol,TResolver](JsonWriter`1& writer, T value, IJsonFormatter`2 formatter, Int32 nextNestingLimit) in C:\projects\spanjson\SpanJson\Formatters\BaseFormatter.cs:line 64
   at SpanJson.Formatters.DictionaryFormatter`4.Serialize(JsonWriter`1& writer, TDictionary value, Int32 nestingLimit) in C:\projects\spanjson\SpanJson\Formatters\DictionaryFormatter.cs:line 56
   at SpanJson.JsonSerializer.Generic.Inner`3.InnerSerializeAsync(T input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 75
   at SpanJson.JsonSerializer.Generic.Utf8.SerializeAsync[T,TResolver](T input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 454
   at SpanJson.JsonSerializer.Generic.Utf8.SerializeAsync[T](T input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.Generics.cs:line 427
   at WebApplication5.Startup.<>c.<<Configure>b__1_1>d.MoveNext() in c:\users\dmytro.kushnir\source\repos\WebApplication5\WebApplication5\Program.cs:line 57
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

I've tried to reduce repro size, but it looks like 16k is some sort of magic number here. If keys are replaced by ascii chars, all is good for much larger cases.

Repro Code:

using System.Collections.Generic;

namespace WebApplication5
{
    public class Data
    {
        public static readonly Dictionary<string, object> Dictionary = new Dictionary<string, object>
        {
            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = new Dictionary<string, object>
            {
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = new Dictionary<string, object>
                {
                    
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                    
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, object>
                    {
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                       
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                    },
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                },
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
            },
            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
            {
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, object>
                {
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                    {
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                        {
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                            {
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                                {
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, object>
                                    {
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                                        {
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                                            {
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                                                {
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                                                    {
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, object>
                                                        {
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                                                            {
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = new Dictionary<string, object>
                                                                {
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                                                                    {
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = new Dictionary<string, object>
                                                                        {
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, object>
                                                                            {
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] =
                                                                                    new Dictionary<string, object>
                                                                                    {
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] =
                                                                                            new Dictionary<string,
                                                                                                object>
                                                                                            {
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] =
                                                                                                    new Dictionary<
                                                                                                        string, object>
                                                                                                    {
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] =
                                                                                                            new
                                                                                                                Dictionary
                                                                                                                <string,
                                                                                                                    int>
                                                                                                                {
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"]
                                                                                                                        = 123,
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"]
                                                                                                                        = 123,
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"]
                                                                                                                        = 123,
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"]
                                                                                                                        = 123,
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"]
                                                                                                                        = 123,
                                                                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"]
                                                                                                                        = 123
                                                                                                                },
                                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                                                    },
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                                            },
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                                    },
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                            },
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                        },
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, int>
                                                                        {
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                        },
                                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                    },
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                                },
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                            },
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                        },
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                    },
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, int>
                                                    {
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                    },
                                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                                },
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                            },
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                        },
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                    },
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                },
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = new Dictionary<string, int>
                                {
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!0"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!1"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!2"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                                },
                                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                            },
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                        },
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                        ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                    },
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!4"] = 123,
                    ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
                },
                ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
            },
            ["ะŸั€ะธะฒะตั‚ ะผะธั€!3"] = 123,
            
            ["ะŸั€ะธะฒะตั‚ ะผะธั€!5"] = 123
        };
    }
}

Main.cs

using System.Net;
using System.Threading.Tasks;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using SpanJson;

namespace WebApplication5
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) => WebHost.CreateDefaultBuilder(args).UseStartup<Startup>();
    }

    public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.Use(next => conext =>
            {
                if (conext.Request.Path == "/favicon.ico")
                {
                    conext.Response.StatusCode = (int) HttpStatusCode.NotFound;
                    return Task.CompletedTask;
                }
                else
                {
                    conext.Response.Headers.Remove("X-Powerd-By");
                    conext.Response.Headers.Remove("X-SourceFiles");
                    return next(conext);
                }
            });

            app.Run(async context =>
            {
                context.Response.ContentType = "application/json";
                await JsonSerializer.Generic.Utf8.SerializeAsync(Data.Dictionary, context.Response.Body);
            });
        }
    }
}

csproj

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="SpanJson" Version="1.0.5" />
  </ItemGroup>

</Project>

Evaluate wrappers for JSON.NET attributes

JSON.NET has a fairly extensive list of attributes, it might be useful as an extra nuget package to map some functionality of these attributes to the functionality of SpanJson.

Candidates

  • JsonPropertyAttribute
  • JsonIgnoreAttribute
  • JsonConstructorAttribute

etc.

Serializing flags enum as string incorrectly includes default (zero) value

For flags, the default value should only be serialized if the value is zero, not if any other flags are set.

[Flags]
public enum FlagsEnum : byte
{
    None = 0b___00000000,
    First = 0b__00000001,
    Second = 0b_00000010,
    Third = 0b__00000100
}

FlagsEnum.None.ToString() == "None" // correct
FlagsEnum.First.ToString() == "First" // correct

JsonSerializer.Generic.Utf16.Serialize(FlagsEnum.None) == "\"None\"" // correct
JsonSerializer.Generic.Utf16.Serialize(FlagsEnum.First) == "\"None,First\"" // incorrect
JsonSerializer.Generic.Utf16.Serialize(FlagsEnum.First | FlagsEnum.Second) == "\"None,First,Second\"" // incorrect

Incorrect flags enum serialization

Code

public class Item
{
    public MyEnum Enum { get; set; }
}

[Flags]
public enum MyEnum
{
    One = 1,
    Two = 2
}
class Program
{
    static void Main(string[] args)
    {
        var item = new Item();

        var spanJson = JsonSerializer.Generic.Utf16.Serialize(item);
        var json = JsonConvert.SerializeObject(item);
    }
}

spanJson

{"Enum":""}

json

{"Enum":0}

I think that Newtonsoft does more correctly, because the value 0 is also a flag.

Array not returned to the array pool when an exception is thrown during serialization

When an exception is thrown during serialization, the rented array is never returned to the array pool.

My suggestion to fix this:

  1. Make JsonWriter<TSymbol>.Dispose() public. Don't make it internal, because others who use the type should also be able to call Dispose().
  2. Remove the implicit dispose in JsonWriter<TSymbol>.ToString().
  3. Use try-finally everywhere JsonWriter<TSymbol> is used and call Dispose() in the finally block.

Redundant quotes, when deserializing dictionary

Example:

JsonSerializer.Generic.Utf8.Deserialize<Dictionary<string, object>>(System.Text.Encoding.UTF8.GetBytes(@"{""a"": ""1""}"))["a"].ToString();

Expected result:
"1"
Actual result:
"\"1\""

Add Unsafe Api to directly return ArrayPool array

The following API could be added to return the array from the arraypool directly, instead of making a copy first.
This API will help cases where the caller might continue working with that array, e.g. copy/write it somewhere else.
The caller will need to return the array to the arraypool himself.

public static ArraySegment<byte> SerializeUnsafe<T>(T value); // 1. variant
// and others

Which type should be used for the returned array, we need the array, the offset and the length.

  • Span: Afaik, there is no way to get the array back from a span
  • ReadOnlySpan: Afaik, there is no way to get the array back from a span, ReadOnly might not be the correct choice if the caller wants further modify the array.
  • ReadOnlyMemory: There is MemoryMarshal.TryGetArray (which returns an ArraySegment), ReadOnly might not be the correct choice if the caller wants further modify the array.
  • ArraySegment: Best fit in my opinion.

Any better ideas?

Can the Formater registration by DependencyInjection or JsonCustomSerializer?

DependencyInjection can be used to facilitate global substitution
eg:
I want Convert DateTime to long.

public sealed class DateTimeToLongFormatter : ICustomJsonFormatter<DateTime>
    {
        public readonly long MinUnixTimeSeconds = DateTimeOffset.MinValue.ToUnixTimeSeconds();
        public DateTime Deserialize(ref JsonReader<byte> reader)
        {
            return DateTimeOffset.FromUnixTimeSeconds(reader.ReadUtf8Int64()).DateTime;
        }
        public DateTime Deserialize(ref JsonReader<char> reader)
        {
            return DateTimeOffset.FromUnixTimeSeconds(reader.ReadUtf8Int64()).DateTime;
        }
        public void Serialize(ref JsonWriter<byte> writer, DateTime value, int nestingLimit)
        {
            writer.WriteUtf8Int64(value <= DateTimeOffset.MinValue.UtcDateTime
                       ? MinUnixTimeSeconds
                       : new DateTimeOffset(DateTime.SpecifyKind(value, DateTimeKind.Utc)).ToUnixTimeSeconds());
        }
        public void Serialize(ref JsonWriter<char> writer, DateTime value, int nestingLimit)
        {
            writer.WriteUtf8Int64(value <= DateTimeOffset.MinValue.UtcDateTime
                      ? MinUnixTimeSeconds
                      : new DateTimeOffset(DateTime.SpecifyKind(value, DateTimeKind.Utc)).ToUnixTimeSeconds());
        }
    }

Add more custom serialization behaviour

It is useful to override default Formatter behaviour of members by adding custom attributes with a formatter to it.
Following changes occur fairly often

  • Change member name: [DataMember] -> already done
  • Ignore member: [IgnoreDataMember] or ShouldSerializeXXX -> already done
  • Change enum value: [EnumMember] -> already done
  • Change the behaviour of the serialization, e.g. write custom serializers for specific types: TODO

FeatureRequest: JsonCustomSerializer with parameters

Hi @Tornhoof ,

Im looking for new JsonSerializer with better performance, which use new stuff in .NetCore 2.1 like Span<T> and Memory<T>.
It's really impressed me about SpanJson's performance. Now im trying to move to SpanJson for my proj.
But during migration from Json.NET to SpanJson there is a feature missing.
With Json.NET I able to configure property converter with parameters like below

public class User
{
     [JsonConverter(typeof(DateTimeConverter), "yyyy-MM-dd HH:mm")]
     public DateTime DateOfBirth {get;set;}
}

How do you think about this feature ?

Why SpanJson.AspNetCore.Formatter require empty constructor?

action

[HttpGet]
public async Task<IList<Group>> Get(CancellationToken ct)
{
    var items = await _db.Query<Group>().AllAsync(ct);
    return items;
}

Group

[UseDto(typeof(GroupDto))]
public class Group : Entity
{
    public Group(string name, string info = null)
    {
        Name = name;
        Info = info;
    }

    public string Name { get; }

    public string Info { get; }
}

Error

An unhandled exception occurred while processing the request.
ArgumentException: Type 'Core.Models.Group' does not have a default constructor
Parameter name: type

System.Linq.Expressions.Expression.New(Type type)
TypeInitializationException: The type initializer for 'SpanJson.Formatters.ComplexClassFormatter`3' threw an exception.

System.Linq.Expressions.Expression.New(Type type)
TargetInvocationException: Exception has been thrown by the target of an invocation.

System.RuntimeFieldHandle.GetValue(RtFieldInfo field, object instance, RuntimeType fieldType, RuntimeType declaringType, ref bool domainInitialized)
TypeInitializationException: The type initializer for 'SpanJson.Formatters.ListFormatter`4' threw an exception.

System.RuntimeFieldHandle.GetValue(RtFieldInfo field, object instance, RuntimeType fieldType, RuntimeType declaringType, ref bool domainInitialized)
TargetInvocationException: Exception has been thrown by the target of an invocation.

System.RuntimeFieldHandle.GetValue(RtFieldInfo field, object instance, RuntimeType fieldType, RuntimeType declaringType, ref bool domainInitialized)
TypeInitializationException: The type initializer for 'Inner`3' threw an exception.

lambda_method(Closure , object , Stream , CancellationToken )

SpanJson.AspNetCore.Formatter == only serialize to string. So why we need empty contructor?

Silent overflow

Example:

class A { public ushort id; }
var result = JsonSerializer.Generic.Utf8.Deserialize<A>(System.Text.Encoding.UTF8.GetBytes(@"{""id"":717036}"));

Expected result:
JsonParserException with inner OverflowException thrown

Actual result:
Executed successfully.
result.id == 61676

SerializeAsync System.Linq.Enumerable.SelectListIterator to stream error

at SpanJson.JsonWriter1.ThrowArgumentException(String message, String paramName) in C:\projects\spanjson\SpanJson\JsonWriter.cs:line 379 at SpanJson.JsonWriter1.WriteUtf8Double(Double value) in C:\projects\spanjson\SpanJson\JsonWriter.Utf8.cs:line 150
at lambda_method(Closure , JsonWriter1& , <>f__AnonymousType1011 )
at SpanJson.Formatters.ComplexClassFormatter3.Serialize(JsonWriter1& writer, T value) in C:\projects\spanjson\SpanJson\Formatters\ComplexClassFormatter.cs:line 33
at SpanJson.Formatters.BaseFormatter.SerializeRuntimeDecisionInternal[T,TSymbol,TResolver](JsonWriter1& writer, T value, IJsonFormatter2 formatter) in C:\projects\spanjson\SpanJson\Formatters\BaseFormatter.cs:line 40
at SpanJson.Formatters.EnumerableFormatter4.Serialize(JsonWriter1& writer, TEnumerable value) in C:\projects\spanjson\SpanJson\Formatters\EnumerableFormatter.cs:line 57
at SpanJson.Formatters.RuntimeFormatter2.Serialize(JsonWriter1& writer, Object value) in C:\projects\spanjson\SpanJson\Formatters\RuntimeFormatter.cs:line 35
at lambda_method(Closure , JsonWriter1& , ActionResult ) at SpanJson.Formatters.ComplexClassFormatter3.Serialize(JsonWriter1& writer, T value) in C:\projects\spanjson\SpanJson\Formatters\ComplexClassFormatter.cs:line 33 at lambda_method(Closure , Object , Stream , CancellationToken ) at SpanJson.JsonSerializer.NonGeneric.Inner2.InnerSerializeAsync(Object input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.NonGeneric.cs:line 120
at SpanJson.JsonSerializer.NonGeneric.Utf8.SerializeAsync(Object input, Stream stream, CancellationToken cancellationToken) in C:\projects\spanjson\SpanJson\JsonSerializer.NonGeneric.cs:line 521

the object can serialize in Newtonsoft.Json

Reducing memory usage for large objects

Introduction
SpanJson reads/writes the data only as a block and if the data to be read/written is large enough to allocate buffers on the LOH the performance and memory usage will suffer.
As long as SpanJson is not working with large arrays on the LOH the GC is happy and we're happy.

Solution
For the streaming async API (Stream/Textwriter/Textreader) a way needs to be found to flush the data at specific points to make sure we don't allocate on the LOH and instead reuse a smaller buffer again.

Writing

  • We have specific points where we can flush, specifically e.g. after an array, list, object etc.
  • It's not necessary to flush after most normal data types, only string and related writer methods
  • We already track the size of the fully serialized/deserialized object, we can use that to decide if we want to run the flush approach or the normal block approach.

POC for writing

I've implemented that idea for writing a large string list, with N "Hello World and Universe" strings.

Questions

  • How often do we flush?
    Obviously before we reach the end of the buffer, this needs some calculations for string et. al.

I also guess it's a function based on buffer size and underlying stream performance,
my current assumption is flush shortly before we reach the LOH size (e.g.~85kb).

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Xeon CPU E5-1620 0 3.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.301
  [Host]     : .NET Core 2.1.1 (CoreCLR 4.6.26606.02, CoreFX 4.6.26606.05), 64bit RyuJIT
  DefaultJob : .NET Core 2.1.1 (CoreCLR 4.6.26606.02, CoreFX 4.6.26606.05), 64bit RyuJIT

Method Count Mean Error StdDev Median Gen 0 Gen 1 Gen 2 Allocated
NewWayUtf16 1000 82.70 us 2.1447 us 3.1436 us 81.06 us 1.7090 - - 9.23 KB
NewWayUtf8 1000 101.63 us 1.2475 us 1.1669 us 101.20 us 0.7324 - - 4.02 KB
OldWayUtf16 1000 77.22 us 0.2569 us 0.2403 us 77.13 us 12.4512 - - 64.02 KB
OldWayUtf8 1000 106.23 us 0.4006 us 0.3345 us 106.14 us 6.2256 - - 32.02 KB
NewWayUtf16 10000 796.14 us 2.0300 us 1.8989 us 795.83 us 0.9766 - - 9.23 KB
NewWayUtf8 10000 1,010.74 us 11.9164 us 11.1466 us 1,005.98 us - - - 4.02 KB
OldWayUtf16 10000 924.31 us 6.7135 us 5.2415 us 923.75 us 332.0313 332.0313 332.0313 1024.02 KB
OldWayUtf8 10000 1,124.74 us 1.5156 us 1.2656 us 1,124.38 us 166.0156 166.0156 166.0156 512.02 KB

custom serializer example

simplified json example

{
   "num_results": 3,
   "more_available": false,
   "status": "ok"
 }

and parse it with custom serializer
as I understand it, I can't do something like that

internal class ResponseDataConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(MyResponse);
    }

    public override object ReadJson(JsonReader reader,
        Type objectType,
        object existingValue,
        JsonSerializer serializer)
    {
        var token = JToken.Load(reader);
        var resp= token.ToObject<MyResponse>();
        var items = token["ext_items"];
       // combine resp and items with complex rules
        ....

and must use ONLY "flow" parsing, right? But how?

    public MyResponse Deserialize(ref JsonReader<char> reader)
    {
        if (reader.ReadIsNull())
        {
            return null;
        }
        reader.ReadBeginObjectOrThrow();
        var name = reader.ReadEscapedName(); //num_results
        var val = reader.ReadInt32(); //3

        //reader.ReadNextToken();

        var name1 = reader.ReadEscapedName(); //SpanJson.JsonParserException: 'Error Reading JSON data: 'ExpectedDoubleQuote' at position: '22'.'
        var val2 = reader.ReadInt32(); 
       ...
    }

I can't read next value: "more_available": false,

Support for something like [JsonExtensionDataAttribute]

Json.NET supports [JsonExtensionData] on an IDictionary<string,object>.
During serialization all values in that dictionary are serialized as additional attributes in the JSON.
During deserialization all unknown values are deserialized into that property.

This is a nice way to allow cheap attribute extensions of objects and a catch-all for unknown properties.

RegisterGlobalCustomFormatter and arrays

Hi,
Is there a way to implement ICustomJsonFormatter that would serialize/deserialize arrays in custom way and register it via
RegisterGlobalCustomFormatter ? I did that without issues for custom enum formatter, but had not luck with arrays.

DateTimeParser doesn't handle negative offsets well - the sign of the minutes is not set correctly.

offsetHours = offsetChar == (byte) '-' ? -offsetHours : offsetHours;
var timeSpan = new TimeSpan(offsetHours, offsetMinutes, 0);

On line 345 the negative sign of the hours is correctly handled but the same thing must be done for the minutes as well. Try serializing and then deserializing the following:

new DateTimeOffset( 2016, 3, 5, 15, 57, 30, TimeSpan.FromMinutes( -90 ) )

When deserialized offset minutes will be 30 and offset hours will be -1 which will expand into:

new TimeSpan( -1, 30, 0 );

Which in turn basically means "00:30:00". I am happy to submit a pull request if you confirm the issue.

[question] Working with enums

Hi,
First of all, thanks for open sourcing this lib, works great (it's really fast)!
I've got couple of questions regarding deserializing enums.
When deserializing from enum to string, would it be possible to deserialize from camelCase strings as well, even when enum is PascalCase? Currently I'm using EnumMember , but was wondering if there is a different way (like in case of camelCaseResolvers).

Another thing I'd like to ask is about deserializing to enum when string value doesn't match any (unknown enum value currently throws). In my use case I know all enum values, but it's a constant stream of data that may introduce new enum value which will currently break my deserialization logic. Would writing custom resolver for that be the best way? I'd like to map such unknown value to default(myenum).
Thanks!

Faster UTF8 Attribute name writing

Problem
Currently the ComplexObjectSerializer uses an Expression.Constant(byte-array) to write the attribute name (incl. quotes and separator) directly, this is quite a bit slower than the string version for utf16.

Idea
Use a similar method as for the deserialization, e.g. write ulong,ushort,byte instead of the full array, as those integer values are then constants which are directly compiled into jit asm constants and no field lookup to the array is necessary.

Benchmarks
Should improve attribute name writing in the utf8 case by ~10ns (from ~20ns for an 8 char name).

Custom formatter not called

I don't understand. I create new dotnetcore 2.2 project (cli) and copy code from test

[JsonCustomSerializer(typeof(TwcsCustomSerializer))]
public class TypeWithCustomSerializer : IEquatable<TypeWithCustomSerializer>
{
    public long Value { get; set; }

    public bool Equals(TypeWithCustomSerializer other)
    {
        if (ReferenceEquals(null, other)) return false;
        if (ReferenceEquals(this, other)) return true;
        return Value == other.Value;
    }

    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj)) return false;
        if (ReferenceEquals(this, obj)) return true;
        return obj is TypeWithCustomSerializer twcs && Equals(twcs);
    }

    public override int GetHashCode()
    {
        // ReSharper disable once NonReadonlyMemberInGetHashCode
        return Value.GetHashCode();
    }
}

public sealed class TwcsCustomSerializer : ICustomJsonFormatter<TypeWithCustomSerializer>
{
    public static readonly TwcsCustomSerializer Default = new TwcsCustomSerializer();

    public object Arguments { get; set; }

    private void SerializeInternal<TSymbol>(ref JsonWriter<TSymbol> writer, TypeWithCustomSerializer value)
        where TSymbol : struct
    {
        if (value == null)
        {
            writer.WriteNull();
            return;
        }


        writer.WriteBeginObject();

        writer.WriteName(nameof(TypeWithCustomSerializer.Value));

        writer.WriteInt64(value.Value);

        writer.WriteEndObject();
    }

    public void Serialize(ref JsonWriter<byte> writer, TypeWithCustomSerializer value)
    {
        SerializeInternal(ref writer, value);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    private TypeWithCustomSerializer DeserializeInternal<TSymbol>(ref JsonReader<TSymbol> reader)
        where TSymbol : struct
    {
        if (reader.ReadIsNull())
        {
            return null;
        }

        reader.ReadBeginObjectOrThrow();
        var result = new TypeWithCustomSerializer {Value = reader.ReadInt64()};
        reader.ReadEndObjectOrThrow();
        return result;
    }

    public TypeWithCustomSerializer Deserialize(ref JsonReader<byte> reader)
    {
        return DeserializeInternal(ref reader);
    }

    public void Serialize(ref JsonWriter<char> writer, TypeWithCustomSerializer value)
    {
        SerializeInternal(ref writer, value);
    }

    public TypeWithCustomSerializer Deserialize(ref JsonReader<char> reader)
    {
        return DeserializeInternal(ref reader);
    }
} 

internal class Program
{
    private static void Main(string[] args)
    {
        var serialized = "{\"Value\":100}";
        var obj = JsonSerializer.Generic.Utf16.Deserialize<TypeWithCustomSerializer>(serialized);
        Debug.Assert(obj.Value == 100);
    }
}

And set breakpoint to all methods. And no any breakpoint was hit.
Also I replace all methods to throw new NotImplementedException(); and I still get successful deserialization

Improve JsonWriter<TSymbol>.WriteUtf(8|16)String methods

Suggestion

Change WriteUtf8String and WriteUtf16String methods signature from:

public void WriteUtf8String(string value)

to

public void WriteUtf8String(ReadOnlySpan<char> value)

Reason

I have a complicated object graph that can not be handled by ComplexFormatter in the way i want it, so i implement ICustomJsonFormatter<T> to handle it.
Inside my custom formatter i need to write ReadOnlySpan<char> onto writer buffer.
Currently, the only way to do that, is to allocate new string, which is not acceptable for me, because is causes GC pauses on hi load.

In my local copy of SpanJson i did that with just a few lines changed and it seems it works fine.

Support for async

async support is a prerequisite for streaming support (especially for utf8) with System.IO.Pipelines

Current challenges

  • Span can't be used in async methods
  • ref-style types can't be passed into async methods

Open questions

  • Serialization: Instead of growing the buffer until everything is serialized, how to flush the current content to the underlying stream if it is full
  • Deserialization: How to wait/obtain new data if the current buffer does not contain all necessary information and how to abort if no data is available.

Ideas

  • Serialization: Flush the data only after a block (e.g. after one object), that should keep both flushes in check and maybe enable us to get around the async method limitations
  • Deserialization: How to wait for new data, only way I currently see is having some kind of sax style parser (like ReadDynamic)

Error deserializing ReadOnlySpan: TypeLoadException: 'The generic type 'SpanJson.Helpers.RecursionCandidate`1' was used with an invalid instantiation in assembly 'SpanJson ...'

Example:

struct A
{
	public string X;

	[IgnoreDataMember]
	public ReadOnlySpan<char> SubX
	{
		get
		{
			return X.Substring(2);
		}
	}
}
var result = JsonSerializer.Generic.Utf8.Deserialize<A>(System.Text.Encoding.UTF8.GetBytes(@"{""X"":""001"", ""SubX"":""2""}"));

Expected:
SubX is marked with [IgnoreDataMember] and should not affect anything

Actual:
System.TypeLoadException: 'The generic type 'SpanJson.Helpers.RecursionCandidate`1' was used with an invalid instantiation in assembly 'SpanJson, Version=1.3.0.0, Culture=neutral, PublicKeyToken=12740fa6726bc6d3'.'

Without SubX deserialization runs successfully

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.