Code Monkey home page Code Monkey logo

brotlisharplib's Introduction

BrotliSharpLib

BrotliSharpLib is a full C# port of the brotli library/compression code by Google. It is intended to be a mostly 1:1 conversion of the original C code. All code is correct as of v0.6.0 of the reference implementation.

The projects uses a minimal set of APIs to ensure compatibility with a wide range of frameworks including .NET Standard and .NET Core. It also supports little-endian and big-endian architectures and is optimised for x86, x64 and ARM processors.

BrotliSharpLib is licensed under MIT.

Installation

BrotliSharpLib can be installed via the NuGet package here.

Install-Package BrotliSharpLib

Usage

Generic/basic usage:

/** Decompression **/
byte[] brotliCompressedData = ...; // arbritary data source
byte[] uncompressedData = Brotli.DecompressBuffer(brotliCompressedData, 0, brotliCompressedData.Length /**, customDictionary **/);

/** Compression **/
byte[] uncompressedData = ...; // arbritary data source

// By default, brotli uses a quality value of 11 and window size of 22 if the parameters are omitted.
byte[] compressedData = Brotli.CompressBuffer(uncompressedData, 0, uncompressedData.Length /**, quality, windowSize, customDictionary **/);

Stream usage:

/** Decompression **/
using (var ms = new MemoryStream())
using (var bs = new BrotliStream(compressedStream, CompressionMode.Decompress))
{
    bs.CopyTo(ms);
}

/** Compression **/
using (var fs = File.OpenRead(filePath))
using (var ms = new MemoryStream())
{
    using (var bs = new BrotliStream(ms, CompressionMode.Compress))
    {
        // By default, BrotliSharpLib uses a quality value of 1 and window size of 22 if the methods are not called.
        /** bs.SetQuality(quality); **/
        /** bs.SetWindow(windowSize); **/
        /** bs.SetCustomDictionary(customDict); **/
        fs.CopyTo(bs);
    }

    byte[] compressed = ms.ToArray();
}

Real-life example: The following allows for acceptance and decompression of brotli encoded web content via a HttpClient and falls back to gzip or deflate when required.

static class HttpClientEx
{
    private class BrotliCompressionHandler : DelegatingHandler
    {
        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("br"));
            var response = await base.SendAsync(request, cancellationToken);
            IEnumerable<string> ce;
            if (response.Content.Headers.TryGetValues("Content-Encoding", out ce) && ce.First() == "br")
            {
                var buffer = await response.Content.ReadAsByteArrayAsync();
                response.Content = new ByteArrayContent(Brotli.DecompressBuffer(buffer, 0, buffer.Length));
            }
            return response;
        }
    }

    public static HttpClient Create()
    {
        var handler = new HttpClientHandler();
        if (handler.SupportsAutomaticDecompression)
            handler.AutomaticDecompression = System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.GZip;
        return HttpClientFactory.Create(handler, new BrotliCompressionHandler());
    }
}

Performance

Considerations for Build

For optimal performance, ensure to build BrotliSharpLib in Release mode to enable all possible JIT optimisations.

Performance can also be further improved by building BrotliSharpLib using .NET Framework 4.5 or above (or any framework that supports AggressiveInlining). Selecting a specific target platform (instead of AnyCPU) where possible can also further improve performance. All of this however, is completely optional as BrotliSharpLib is designed to run in a wide range of contexts and configurations regardless.

Benchmark

The following are benchmark results using DotNetBenchmark with BrotliSharpLib (v0.2.1) and Google's C# implementation built against .NET Framework 4.6.1. The original C version was compiled in Release mode using Visual Studio 2017 (v141) as a 64-bit Windows executable.

BenchmarkDotNet=v0.10.6, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i5-6600K CPU 3.50GHz (Skylake), ProcessorCount=4
Frequency=3421875 Hz, Resolution=292.2374 ns, Timer=TSC
  [Host]       : Clr 4.0.30319.42000, 64bit RyuJIT-v4.7.2046.0
  RyuJitX64    : Clr 4.0.30319.42000, 64bit RyuJIT-v4.7.2046.0
Runtime=Clr  

Decompression

File: UPX v3.91 (Windows Executable)

Method Mean
GoogleImpl 12.75 ms
BrotliSharpLib 11.63 ms
Original C 11.17 ms

As seen above, BrotliSharpLib performs close to the original C version in terms of decompression.

Compression

File: plrabn12.txt

Method Quality Mean
BrotliSharpLib 1 9.132 ms
Original C 1 9.570 ms
BrotliSharpLib 6 58.720 ms
Original C 6 36.540 ms
BrotliSharpLib 9 116.318 ms
Original C 9 73.080 ms
BrotliSharpLib 11 1822.702 ms
Original C 11 877.58 ms

While BrotliSharpLib performs comparatively at lower quality levels, it performs up to three times worse at level 11. Future versions of the port will hopefully bring this down.

brotlisharplib's People

Contributors

egorbo avatar kfreezen avatar master131 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

brotlisharplib's Issues

How do those improvements work?

Hello,
I was looking through the commits and noticed two commits: 1c452a9 and 429a82a

The first one changed some code, and I'd like to ask what changes were made here, and more importantly how exactly that improved performance? What was the big bottleneck here?
I understand the code (at least somewhat), but the diffs are a bit mangled and hard to follow.

The second one surprises me as well, how were those aggressive inlining attributes identified as harmful to the performance? Did you manually comment them out and re-run the benchmarks? (I doubt it a bit, that'd be a somewhat "blind" approach, no??) Did you inspect the generated asm code and identified some issues there?

Great work on the library! I love it! :)

Feature Request: Change default quality level to 1

Since Quality 11 is super-slow (especially in c# version), it seems like it should be opt-in instead of default. Most applications will need to make a trade off of speed vs size. The difference in size between Quality 1 and 11 isn't very much (~ 5%), but the speed difference is astronomical (~ 100x slower for C or 200x slower for c#).

It seems unlikely anyone will want to use quality 11. The majority will probably be best served by quality 1.
It might be good to change the default for people who don't read the documentation thoroughly enough & accidentally slow down their application to a halt.

P.S. This is a great lib. Thank you!

Error -11, -9

Hi,

I have trouble to decompress my protobuf data within a netcore v2 project. I haven't tested if this problem is related to protobuf encoded data only but when I use the same stream in combination with deflate, everything works fine.

I have tested the default quality (which is 11 as far as I know, and 3). By using the quality level 11 I get a -11 error and with a quality level of 3 I get a -9 error.

Compressing the data seems to work well.

Decompress:
//using (var decomressor = new BrotliStream(source, CompressionMode.Decompress)) using (var decomressor = new DeflateStream(source, CompressionMode.Decompress)) { return Serializer.Deserialize<T>(decomressor); }

Compress:

//using (var compressor= new BrotliStream(destination, CompressionMode.Compress)) using (var compressor = new DeflateStream(destination, CompressionMode.Compress)) { //compressor.SetQuality(3); Serializer.Serialize(compressor, data); }

Sorry for the bad format of the code. I don't know why this is printed that stupid in the preview?!

I have attached the 2 brotli compressed files with quality level 3 and 11.
files.zip

DecompressionViaStream fails on .NET Core 2.0 and above.

DecompressViaStream test currently fails on .NET Core 2.x/3.x but passes on .NET Core 1.x.

Message: System.Exception : Decompress failed with for mapsdatazrh.compressed
----> System.ArgumentException : Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.

OutofMemory Exception

Every now and then we get following error. Any fix that may be available to resolve this?

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb)
at BrotliSharpLib.Brotli.DefaultAllocFunc(Void* opaque, SizeT size)
at BrotliSharpLib.Brotli.BrotliAllocate(MemoryManager& m, SizeT n)
at BrotliSharpLib.Brotli.RingBufferInitBuffer(MemoryManager& m, UInt32 buflen, RingBuffer* rb)
at BrotliSharpLib.Brotli.RingBufferWrite(MemoryManager& m, Byte* bytes, SizeT n, RingBuffer* rb)
at BrotliSharpLib.Brotli.CopyInputToRingBuffer(BrotliEncoderStateStruct& s, SizeT input_size, Byte* input_buffer)
at BrotliSharpLib.Brotli.BrotliEncoderCompressStream(BrotliEncoderStateStruct& s, BrotliEncoderOperation op, SizeT* available_in, Byte** next_in, SizeT* available_out, Byte** next_out, SizeT* total_out)
at BrotliSharpLib.BrotliStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count, BrotliEncoderOperation operation)
at BrotliSharpLib.BrotliStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at WebMarkupMin.AspNet4.Common.HttpCompressionFilterStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at WebMarkupMin.AspNet4.Common.MarkupMinificationFilterStream.Close()
at System.Web.HttpWriter.FilterIntegrated(Boolean finalFiltering, IIS7WorkerRequest wr)
at System.Web.HttpResponse.FilterOutput()
at System.Web.HttpApplication.CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Strong named assembly

Can we release a strongly named nuget package, so that other packages that are strongly named can use this. For now in Titanium Web Proxy someone have published a strongly named version of this library. Not sure if its official.

BrotliStream quality levels from 5 to 9 don't work on NetCore.

That exception is throwed on Quality levels 5,6,7,8 and 9

System.NullReferenceException: Object reference not set to an instance of an object.

StackTrace:
at BrotliSharpLib.Brotli.ShouldUseComplexStaticContextMap(Byte* input, SizeT start_pos, SizeT length, SizeT mask, Int32 quality, SizeT size_hint, ContextType* literal_context_mode, SizeT* num_literal_contexts, UInt32** literal_context_map)
at BrotliSharpLib.Brotli.DecideOverLiteralContextModeling(Byte* input, SizeT start_pos, SizeT length, SizeT mask, Int32 quality, SizeT size_hint, ContextType* literal_context_mode, SizeT* num_literal_contexts, UInt32** literal_context_map)
at BrotliSharpLib.Brotli.WriteMetaBlockInternal(MemoryManager& m, Byte* data, SizeT mask, UInt64 last_flush_pos, SizeT bytes, Boolean is_last, BrotliEncoderParams* params_, Byte prev_byte, Byte prev_byte2, SizeT num_literals, SizeT num_commands, Command* commands, Int32* saved_dist_cache, Int32* dist_cache, SizeT* storage_ix, Byte* storage)
at BrotliSharpLib.Brotli.EncodeData(BrotliEncoderStateStruct& s, Boolean is_last, Boolean force_flush, SizeT* out_size, Byte** output)
at BrotliSharpLib.Brotli.BrotliEncoderCompressStream(BrotliEncoderStateStruct& s, BrotliEncoderOperation op, SizeT* available_in, Byte** next_in, SizeT* available_out, Byte** next_out, SizeT* total_out)
at BrotliSharpLib.BrotliStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count, BrotliEncoderOperation operation)
at BrotliSharpLib.BrotliStream.FlushCompress(Boolean finish)
at BrotliSharpLib.BrotliStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()

[Xamarin.iOS] Crash on System.ExecutionEngineException: Attempting to JIT compile method...

I received the following exception when decompressing brotli file with Stream method:


{System.ExecutionEngineException: Attempting to JIT compile method '(wrapper other) void BrotliSharpLib.Brotli/BrotliDecoderStateStruct:PtrToStructure (intptr,object)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.

at (wrapper managed-to-native) System.Runtime.InteropServices.Marshal.PtrToStructure(intptr,System.Type)
at System.Runtime.InteropServices.Marshal.PtrToStructure[T] (System.IntPtr ptr) [0x00000] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/mcs/class/corlib/System.Runtime.InteropServices/Marshal.cs:930
at BrotliSharpLib.Brotli.CreateStruct[T] () [0x00021] in <8abe3495f7f047ed822d11fb9d962483>:0
at BrotliSharpLib.Brotli.BrotliCreateDecoderState () [0x00000] in <8abe3495f7f047ed822d11fb9d962483>:0
at BrotliSharpLib.BrotliStream..ctor (System.IO.Stream stream, System.IO.Compression.CompressionMode mode, System.Boolean leaveOpen) [0x00079] in <8abe3495f7f047ed822d11fb9d962483>:0
at BrotliSharpLib.BrotliStream..ctor (System.IO.Stream stream, System.IO.Compression.CompressionMode mode) [0x00000] in <8abe3495f7f047ed822d11fb9d962483>:0
at ...
at ...
base: {System.SystemException}


First instance spikes with 300 MB memory usage

Consider the following test application (.NET Core):

using System;
using System.IO;

using BSL = BrotliSharpLib;
using System.IO.Compression;

namespace BrotliTest
{

    class Program
    {

        static void Main(string[] args)
        {
            using (var mem = new MemoryStream())
            {
                using (var brotli = new BSL.BrotliStream(mem, CompressionMode.Compress))
                {

                }
            }

            Console.ReadLine();
        }

    }

}

When executed, you will see a 300 MB process memory spike in Visual Studio's Diagnostics tool window. This only happens on first instantiation of a BrotliStream. Is there a way to improve memory management in this regard?

make BrotliSharpLib .netstandard2.0 compatible

i'm using BrotliSharpLib on my project (rest api) targeting .net461 and it works ok. however, I've ported my project to .netstandard2.0 (netcoreapp2.1) and it seems not to be working correctly - serialization of 7000 objects (each of them has 10 string properties) takes approx. 15-16 secs and my endpoint responds in 20sec instead of 3.
Can you please make BrotliSharpLib .netstandard2.0 compatible or investigate what may cause the delay - taking 15secs to serialize 7000 objects doesn't seem to be right?

Tests are failing

Hi,
I've downloaded sources, but tests in test project fail.
Decompress
DecompressViaStream

Message:
Decompressed length does not match original (alice29.txt.compressed --> alice29.txt)
Expected: 148481
But was: 152089

Message:
Decompressed length does not match original (alice29.txt.compressed --> alice29.txt)
Expected: 148481
But was: 152089

Implement a `DisposeAsync` method for the `BrotliStream` class

When using the current implementation of BrotliStream class in ASP.NET Core 3.0 Preview 7 web application you may receive the following error:

InvalidOperationException: Synchronous operations are disallowed.
Call WriteAsync or set AllowSynchronousIO to true instead.

To fix this error, need to implement in the BrotliStream class a DisposeAsync method (included in .NET Standard 2.1).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.