impromptuninjas / zstd Goto Github PK

When using ZStdDecompressStream to decompress the input data, it's Read method always returns zero after the first read. As per the Stream documentation, a zero return value indicates the end-of-stream. Therefore, it becomes impossible to extract the entire decompressed contents.

Bug can be reproduced by the following code

static void Main(string[] args)
{
    var input = new byte[1024 * 4];
    var compressed = Compress(input);
    Console.WriteLine("Input Length: {0}, Compressed Length: {1}", input.Length, compressed.Length);

    var decompressed = Decompress(compressed);
    Console.WriteLine("Compressed Length: {0}, Decompressed Length: {1}", compressed.Length, decompressed.Length);
}

static byte[] Compress(byte[] input)
{
    using var compressed = new MemoryStream();
    using var compressStream = new ZStdCompressStream(compressed);
    compressStream.Write(input, 0, input.Length);
    compressStream.Flush();
    return compressed.ToArray();
}

static byte[] Decompress(byte[] input)
{
    using var compressed = new MemoryStream(input);
    using var decompressStream = new ZStdDecompressStream(compressed);

    using var output = new MemoryStream();
    var buffer = new byte[1024];
    var bytesRead = 0;

    //This while loop runs only once regardless of the length of buffer
    while ((bytesRead = decompressStream.Read(buffer, 0, buffer.Length)) > 0)
        output.Write(buffer, 0, bytesRead);
    
    return output.ToArray();
}

The program output:

Input Length: 4096, Compressed Length: 18
Compressed Length: 18, Decompressed Length: 1024

OS: Windows 10
Platform: .NET 5.0
Visual Studio: 16.10.4
ImpromptuNinjas.ZStd: 1.4.5.5

Would be extreamly useful if able to bring to Unity for ECS Burst. But too many dependencies (expecially Fody) had caused so many issues..

The zstd lib recently stabilized ZDICT_finalizeDictionary() which can be used to create a dictionary with pre-selected contents rather than relying on the default training algorithm. This can be very useful for file types where there is a known repeating sequence that default training does not find.

The below additions should be all that is needed to add support. I would create a PR but was having some git issues due to the repo being over it's LFS transfer limit.

edit: was able to work around LFS issues. PR opened

Hi, in the read method of Extensions.cs:

    internal static int Read(this Stream stream, Span<byte> bytes) {
      var count = bytes.Length;
      var copy = ArrayPool<byte>.Shared.Rent(count);
      try {
        var read = stream.Read(copy, 0, count);
        copy.CopyTo(bytes);

        return read;
      }
      finally {
        ArrayPool<byte>.Shared.Return(copy);
      }
    }

if the length of bytes is not a power of 2, then the rented array copy (which is only guaranteed to be at least count bytes in length) will be bigger than count, and the line copy.CopyTo(bytes) will throw. I believe you can instead

...
copy.AsSpan(0, read).CopyTo(bytes);
...

Hi, I think there's a possible bug when attempting to dispose a (de)compress stream twice.

    protected override unsafe void Dispose(bool disposing) {
      Flush();

      if (disposing && !Disposed) {
        Decompressor?.Dispose();
        Marshal.FreeHGlobal((IntPtr) Input.GetPinnedPointer());
        Disposed = true;
      }
...

Flush() uses Input, which is freed on the first call to Dispose. Dispose should be safe to call twice. I think you can just replace it with

    protected override unsafe void Dispose(bool disposing) {
      if (!Disposed) {
          Flush();
          if (disposing) {
              Decompressor?.Dispose();
              Marshal.FreeHGlobal((IntPtr) Input.GetPinnedPointer());
              Disposed = true;
          }
      }
...

Does this seem sensible?

There is a new version :) https://github.com/facebook/zstd/releases/tag/v1.5.0

I would like to use your library in my project (a cross-platform package manager, called Zero Install). However, it targets .NET Framework 4.5 and uses strong-name signed assemblies.

Could you add strong name signing to your assemblies? The official guidance from Microsoft on topic says strong-names are not usually needed anymore but recommends signing any public libraries for compatibility.

I could provide a small PR for this if you're interested.