gfoidl / base64 Goto Github PK

View Code? Open in Web Editor NEW

57.0 57.0 7.0 1.58 MB

Base64 encoding / decoding with SIMD-support, also base64Url

Home Page: https://gfoidl.github.io/Base64

License: Other

Shell 3.34% C# 96.66%

base64 base64-decoding base64-encoding base64url simd

base64's People

Contributors

Stargazers

Watchers

Forkers

jinqi166 martinnv6 xiaoshzx ycrumeyrolle aigiol cpu4004

base64's Issues

Remove Strings.Designer.cs and use Resources direct

Thus the fake-tets for the ressourcen can be omitted.

Add target for .NET 4.5

There are some projects out which target .NET 4.5, so this project should target that too.

Use 16 byte load for AVX2

According to Intel® 64 and IA-32 Architectures Optimization Reference Manual 11.6.2:

For best results use Intel AVX 32-byte loads and align data to 32-bytes. However, there are cases where you cannot align the data, or data alignment is unknown. .... In these cases, using 16-Byte memory accesses may be the best alternative.

CI should run fuzz-tests per cron-job

E.g. once a week for an hour.
Ideally if any bugs are found, an issue should be created.

[Proposal] Support ReadOnlySequence as input

The Encode / Decode operations support a ReadOnlySpan<byte> as data input.
It would be interesting to also support a ReadOnlySequence<byte> as data input for chunked data blocks.

Move more invariants out of loops

E.g.

Base64/source/gfoidl.Base64/Internal/Base64Encoder/Base64Encoder.Decode.netcoreapp3.0.cs

Line 311 in 8c5d5c7

Vector128<sbyte> zero = Vector128<sbyte>.Zero;

Benchmarks for MacOS are timing out

In CI the benchmarks for MacOS are timing out.

See
https://dev.azure.com/gh-gfoidl/github-Projects/_build/results?buildId=372
https://dev.azure.com/gh-gfoidl/github-Projects/_build/results?buildId=374

Revert Unsafe.WriteUnaligned workaround

dotnet/runtime#11524 got fixed, so Unsafe.WriteUnaligned can be used (for correctness).

E.g.

Base64/source/gfoidl.Base64/Internal/Vector256Helper.cs

Lines 70 to 75 in 90470c2

    
           public static void WriteVector256(this ref byte dest, Vector256<sbyte> vec) 
        
           { 
        
               // As has better CQ than WriteUnaligned 
        
               // https://github.com/dotnet/coreclr/issues/21132 
        
               Unsafe.As<byte, Vector256<sbyte>>(ref dest) = vec; 
        
           }

Search for: https://github.com/dotnet/coreclr/issues/21132

Log hardware info in benchmarks like AVX2 available / supported

On same VMs it may happen that AVX2 is not available, hence the comparison to SSE may by misleading.
Therefore this info should be logged, too. lscpu may be enough.

.NET Standard 2.1 support

At least for encoding to string, .NET Standard 2.1 offers span-based api, so it could be an improvement.

For CI need to figure out a strategy to test this. Mono supports .NET Standard 2.1 so maybe via this route.

Fix attributation / credits

Add comments / links from where the code got inspired.

Add an analyzer to warn for pubternal usage

Base64/source/gfoidl.Base64/Base64.cs

Lines 13 to 14 in 043d093

    
           private static readonly Base64Encoder    s_default = new Base64Encoder(); 
        
           private static readonly Base64UrlEncoder s_url     = new Base64UrlEncoder();

the internal types are made public (in order to allow inlining after devirtualization in the JIT).

This is subject to change, so an analyzer would be nice to warn if a user uses theses types directly.

Base64Encoder.GetDecodedLength may be unsafe if length < 2

Base64/source/gfoidl.Base64/Internal/Base64Encoder/Base64Encoder.Decode.cs

Lines 29 to 30 in 01406a2

    
           if (end == EncodingPad) padding++; 
        
           if (Unsafe.Subtract(ref end, 1) == EncodingPad) padding++;

has no check if the access is safe. I.e. it can read before the begin of the data, and this is a critical bug.

Document exceptions in xml comments

Fuzz-test should focus on decoding

Encoding is "safe" as no input has to be validated.
Decoding, on the other side, has to validate input, so it's more interesting to fuzz-test.

Code coverage sinks due tests only running with AVX

CI runs with hosts that have AVX2 support, so due #124 some SSE paths aren't hit any more.

codecov-reports not shown / uploaded correctly

E.g. see https://dev.azure.com/gh-gfoidl/github-Projects/_build/results?buildId=663
There's no codecov-report.

Benchmarks don't run

E.g. https://dev.azure.com/gh-gfoidl/github-Projects/_build/results?buildId=664&view=logs

--------------------
SIMD-Info
Sse  : True
Sse2 : True
Ssse3: True
Avx  : True
Avx2 : True
--------------------
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore  /p:UseSharedCompilation=false in /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086
// command took 3.68s and exited with 1
// ***** Done, took 00:00:03 (3.82 sec)   *****
// Found benchmarks:
//   Base64EncoderBenchmark.Encode_Data: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.Encode_Guid: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.Decode_Data: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.Decode_Guid: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToEncode: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToDecode: AVX2(Runtime=Core)
//   Base64EncoderBenchmark.Encode_Data: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.Encode_Guid: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.Decode_Data: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.Decode_Guid: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToEncode: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToDecode: SSSE3(EnvironmentVariables=COMPlus_EnableAVX=0, Runtime=Core)
//   Base64EncoderBenchmark.Encode_Data: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)
//   Base64EncoderBenchmark.Encode_Guid: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)
//   Base64EncoderBenchmark.Decode_Data: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)
//   Base64EncoderBenchmark.Decode_Guid: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToEncode: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)
//   Base64EncoderBenchmark.GetArraySizeRequiredToDecode: Scalar(EnvironmentVariables=COMPlus_EnableSSE=0, Runtime=Core)

// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 


// Build Exception: Standard output: 
   Restore completed in 22.19 ms for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/gfoidl.Base64.Benchmarks.csproj.
  Restore completed in 0.86 ms for /home/vsts/work/1/s/source/gfoidl.Base64/gfoidl.Base64.csproj.
/home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project gfoidl.Base64.Benchmarks is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Project gfoidl.Base64.Benchmarks supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Restore failed in 2.62 sec for /home/vsts/work/1/s/perf/gfoidl.Base64.Benchmarks/bin/Release/netcoreapp3.0/f83e977d-ae6a-49dc-af5f-ff3105f35086/BenchmarkDotNet.Autogenerated.csproj.

 
 Standard error: 
 
...

Reduce overhead

As benchmark results show, there's pretty much overhead.

Investigate and reduce. The numbers should be at least on par with the reference implementation.

CI: update MacOS image

CI reports

##[warning]This pipeline uses a Microsoft-hosted agent image that will be removed on March 23, 2020 (MacOS-10.13). You must make changes to your pipeline before that date, or else your pipeline will fail. Learn more (https://aka.ms/removing-older-images-hosted-pools).

So update the image.

CI with AVX2 and / or SSSE3 disabled

...in order to test the different code-path.

Undo coverlet version fix to 1.3.0

coverlet-coverage/coverlet#274 got fixed, so 4487852 can be undone.

Codecov comments should be minimal

Base64UrlEncoder.Decode reuse AVX vectors

For context see #124 (comment)

Due to JIT limits (too many locals) the avx-codepath won't inline. Thus the change got [reverted]
(#124 (comment)).

Recheck later if JIT improved, so this method can get the benefit of reusing the avx vectos and smaller code size.

Undo disabling code-coverage on debug-builds

Undo 439cc74 when the root-issue got fixed.

Cf. #28 (comment)

Rename `sequential` labels to `scalar` labels

Well, the opposite of vectorized is scalar.
The opposite of sequential would be parallel.

So the names are wrong.

Perf regression for encoding 16 bytes on windows

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.17763.737 (1809/October2018Update/Redstone5)
Intel Xeon CPU E5-2673 v3 2.40GHz, 1 CPU, 2 logical and 2 physical cores
.NET Core SDK=3.1.100-preview2-014569
  [Host] : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  AVX2   : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  SSSE3  : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  Scalar : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT

|        Method |    Job | EnvironmentVariables | DataLen |        Mean |     Error |    StdDev | Ratio | RatioSD |
|-------------- |------- |--------------------- |-------- |------------:|----------:|----------:|------:|--------:|
| BuffersBase64 |   AVX2 |                Empty |       5 |    19.32 ns |  0.193 ns |  0.180 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |       5 |    18.57 ns |  0.324 ns |  0.303 ns |  0.96 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |    17.56 ns |  0.187 ns |  0.166 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |    17.23 ns |  0.201 ns |  0.188 ns |  0.98 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |    16.01 ns |  0.218 ns |  0.204 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |    16.01 ns |  0.196 ns |  0.173 ns |  1.00 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |      16 |    23.93 ns |  0.175 ns |  0.164 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |      16 |    33.76 ns |  0.473 ns |  0.442 ns |  1.41 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |    21.10 ns |  0.168 ns |  0.157 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |    34.32 ns |  0.597 ns |  0.558 ns |  1.63 |    0.03 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |    32.69 ns |  0.516 ns |  0.457 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |    28.00 ns |  0.270 ns |  0.225 ns |  0.86 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |    1000 |   133.79 ns |  0.837 ns |  0.783 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |    1000 |   137.12 ns |  1.358 ns |  1.270 ns |  1.02 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 |   216.30 ns |  1.895 ns |  1.772 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 |   220.61 ns |  2.709 ns |  2.534 ns |  1.02 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 1,214.37 ns |  8.696 ns |  8.134 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 1,200.87 ns | 13.526 ns | 12.652 ns |  0.99 |    0.01 |

linux results

BenchmarkDotNet=v0.12.0, OS=ubuntu 18.04
Intel Xeon CPU E5-2673 v4 2.30GHz, 1 CPU, 2 logical and 2 physical cores
.NET Core SDK=3.1.100-preview2-014569
  [Host] : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  AVX2   : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  SSSE3  : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  Scalar : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT

|        Method |    Job | EnvironmentVariables | DataLen |        Mean |     Error |    StdDev | Ratio | RatioSD |
|-------------- |------- |--------------------- |-------- |------------:|----------:|----------:|------:|--------:|
| BuffersBase64 |   AVX2 |                Empty |       5 |    23.20 ns |  0.476 ns |  0.422 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |       5 |    17.82 ns |  0.230 ns |  0.204 ns |  0.77 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |    22.49 ns |  0.448 ns |  0.419 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |    16.38 ns |  0.221 ns |  0.207 ns |  0.73 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |    21.70 ns |  0.268 ns |  0.251 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |    16.04 ns |  0.226 ns |  0.212 ns |  0.74 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |      16 |    24.33 ns |  0.291 ns |  0.243 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |      16 |    23.18 ns |  0.437 ns |  0.409 ns |  0.95 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |    24.90 ns |  0.538 ns |  0.553 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |    21.04 ns |  0.226 ns |  0.211 ns |  0.85 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |    35.38 ns |  0.386 ns |  0.361 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |    28.45 ns |  0.391 ns |  0.365 ns |  0.80 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |    1000 |   140.33 ns |  0.602 ns |  0.533 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |    1000 |   140.82 ns |  1.005 ns |  0.940 ns |  1.00 |    0.01 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 |   223.42 ns |  2.105 ns |  1.866 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 |   217.72 ns |  4.281 ns |  4.581 ns |  0.98 |    0.02 |
|               |        |                      |         |             |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 1,230.42 ns | 21.478 ns | 20.091 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 1,218.58 ns | 24.015 ns | 31.226 ns |  1.00 |    0.02 |

mac results

BenchmarkDotNet=v0.12.0, OS=macOS High Sierra 10.13.6 (17G9016) [Darwin 17.7.0]
Intel Xeon CPU E5-1650 v2 3.50GHz (Max: 3.34GHz), 2 CPU, 4 logical and 4 physical cores
.NET Core SDK=3.1.100-preview2-014569
  [Host] : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  AVX2   : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  SSSE3  : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT
  Scalar : .NET Core 3.1.0 (CoreCLR 4.700.19.52202, CoreFX 4.700.19.52317), X64 RyuJIT

|        Method |    Job | EnvironmentVariables | DataLen |      Mean |     Error |    StdDev | Ratio | RatioSD |
|-------------- |------- |--------------------- |-------- |----------:|----------:|----------:|------:|--------:|
| BuffersBase64 |   AVX2 |                Empty |       5 |  16.40 ns |  0.394 ns |  0.625 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |       5 |  12.24 ns |  0.271 ns |  0.353 ns |  0.75 |    0.03 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |  15.60 ns |  0.273 ns |  0.255 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |       5 |  12.03 ns |  0.259 ns |  0.242 ns |  0.77 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |  15.84 ns |  0.345 ns |  0.369 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |       5 |  11.45 ns |  0.171 ns |  0.160 ns |  0.72 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |      16 |  18.09 ns |  0.381 ns |  0.357 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |      16 |  15.06 ns |  0.297 ns |  0.278 ns |  0.83 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |  18.30 ns |  0.356 ns |  0.333 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |      16 |  16.25 ns |  0.347 ns |  0.324 ns |  0.89 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |  25.59 ns |  0.526 ns |  0.540 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |      16 |  21.21 ns |  0.318 ns |  0.297 ns |  0.83 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 |   AVX2 |                Empty |    1000 | 180.36 ns |  2.034 ns |  1.903 ns |  1.00 |    0.00 |
|  gfoidlBase64 |   AVX2 |                Empty |    1000 | 172.12 ns |  1.826 ns |  1.708 ns |  0.95 |    0.01 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 | 186.66 ns |  2.787 ns |  2.607 ns |  1.00 |    0.00 |
|  gfoidlBase64 |  SSSE3 |  COMPlus_EnableAVX=0 |    1000 | 177.80 ns |  2.551 ns |  2.386 ns |  0.95 |    0.02 |
|               |        |                      |         |           |           |           |       |         |
| BuffersBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 901.81 ns | 15.498 ns | 14.497 ns |  1.00 |    0.00 |
|  gfoidlBase64 | Scalar |  COMPlus_EnableSSE=0 |    1000 | 939.78 ns | 13.885 ns | 12.309 ns |  1.04 |    0.02 |

Linux and Mac don't show this regression.
A local perf-run doesn't show this either.

So far no action is needed, but this issue is to keep an eye on the windows-results.

Partially bad performance for .NET Full

has quite bad perf. This was never investigated so far, only for .NET Core targets.

Undo MacOS-hack / skip for AVX2

The hack from #1 (comment) needs to be undone, i.e. revert 469f256

Drop support for .NET Core 2.1

.NET Core 3.0 is current, so no need for an unsupported runtime in regards for HW-intrinsics. And it just complicates things.

AVX2 decoding slower than SSSE3 decoding

Baseline therefore are the benchmark-results from CI.

Comparison of AVX2 to SSSE3

The table below shows the "winner" of each category.

Benchmark	OS	Encode	Decode
Utf8	Linux	AVX2	SSSE3
Utf8	Windows	AVX2	SSSE3
String	Linux	-	SSSE3
String	Windows	AVX2	SSSE3

For Mac / OSX AVX2 is currently not supported / disabled, so a comparison wouldn't have any value, as it actually is SSSE3 vs SSSE3.

So for encoding AVX2 is (almost) always faster.
For decoding SSSE3 is (almost) always (ways, up to ~2x) faster.

Hardware info

Info is gathered with help of CPU-output in BDN.

OS / CPU	AVX2	SSSE3
Linux / Intel Xeon CPU E5-2673	✔️	✔️
Windows / Intel Xeon CPU E5-2673	✔️	✔️
OSX / Intel Xeon CPU E5-1650		✔️

Base64Url does not detect invalid data accurately

Fuzzing found (after a few seconds)

z\0dYsqEkYYYYYEkYYYYYQYYeQYYeker
pppppppÿÿpppÿÿpppppppp_

which are in the SSSE3-range, but it didn't detect invalid data.

crashes.zip

CI fails for windows-benchmark

https://dev.azure.com/gh-gfoidl/github-Projects/_build/results?buildId=173:

dotnet_install: Error: OS name could not be detected: UName = MINGW64_NT-10.0

Aim towards 100% code coverage

Code coverage is at 98%, so there's a small room for improvement.

Base64.cs is untested at all.
Base64UrlEncoder.GetBufferSizeRequiredToBase64Encode is also untested.

Replace int-math with uint-math

E.g.

Base64/source/gfoidl.Base64/Internal/Base64EncoderImpl.cs

Line 106 in 90470c2

int numWholeOrPartialInputBlocks = (sourceLength + 2) / 3;

With uint the JIT can generate better code (but only where safe).

Tune theshold-constants

E.g.

Base64/source/gfoidl.Base64/Internal/Base64Encoder/Base64Encoder.Encode.netcoreapp.cs

Lines 20 to 23 in 9e4584b

    
           // Threshoulds found by testing -- may not be ideal on all targets 
        
           if (data.Length < 26) 
        
               return Convert.ToBase64String(data);

does still apply?

Also check #95 (comment)

Clean up xml-doc comments with <inheritdoc>

Cf. <inheritdoc>.

Quite some noise-reduction possible in the code.

Use latest SDK for netcoreapp3.0

Right now the SDK version is fixed. Ideal would be latest.

SSE2 reuse lower part of AVX2 vectors

Most (or all?) vectors should be equal in the lower parts, so they can be reused instead of recreated. This would shrink the generated code.

Add xml-comments / documentation

Don't run benchmarks on every commit to master

They shall run per cron-build after a change happened.
E.g. per cron daily, so several changes can be grouped together. Detecting any regression is quite easy, as the range of commits is known.

Add a strong name to the package

In order to allow integration to applications or packages that are themself with a strong name, this package should be signed.
THis is generating the following warning message:
CSC : warning CS8002: Referenced assembly 'gfoidl.Base64, Version=1.0.761.1, Culture=neutral, PublicKeyToken=null' does not have a strong name.

https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/strong-naming

CI does not trigger for forked PRs

Context: #24 (comment)

Annotate with nullable and fix code-styles

Investigate port 5 pressure

According to Intel® 64 and IA-32 Architectures Optimization Reference Manual 11.11

There are a few cases where shuffles such as VSHUFPS or VPERM2F128 can be replaced by blend nstructions. Intel AVX shuffles are executed only on port 5, while blends are also executed on port 0. Therefore, replacing shuffles with blends could reduce port 5 pressure.

Base64/source/gfoidl.Base64/Internal/Avx2Helper.cs

Lines 22 to 23 in 4c8c69f

    
           Vector256<sbyte> t0 = Avx2.Permute2x128(c0, c1, 0x20); 
        
           Vector256<sbyte> t1 = Avx2.Permute2x128(c0, c1, 0x31);

Try to replace both with blend, also try to just replace one with blend.

Provide method to detect whether base64 or base64Url

Sometimes it may not be clear if the encoded data is base64 or base64Url. So a method for that detection would be nice.

MissingMethodException for AVX on windows

Hi,

Just tried the encoder on Windows, but I have this error either when running tests or benchmarks:
Tests:

Message: System.MissingMethodException : Method not found:
System.Runtime.Intrinsics.Vector256`1<!!1>
System.Runtime.Intrinsics.X86.Avx.StaticCast(System.Runtime.Intrinsics.Vector256`1<!!0>).

Benchmarks:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.TypeInitializationException: The type initializer for 'gfoidl.Base64.Base64Encoder' threw an exception. ---> System.MissingMethodException: Method not found: 'System.Runtime.Intrinsics.Vector256`1<SByte> System.Runtime.Intrinsics.X86.Avx.SetVector256(SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte, SByte)'.
   at gfoidl.Base64.Base64Encoder..cctor()
   --- End of inner exception stack trace ---
   at gfoidl.Base64.Base64Encoder..ctor()
   at gfoidl.Base64.Base64.get_Default() in C:\dev\git\Base64\source\gfoidl.Base64\Base64.cs:line 16
   at gfoidl.Base64.Benchmarks.Base64EncoderBenchmarks..ctor() in C:\dev\git\Base64\perf\gfoidl.Base64.Benchmarks\Base64EncoderBenchmarks.cs:line 5
   at BenchmarkDotNet.Autogenerated.Runnable_17..ctor() in C:\dev\git\Base64\perf\gfoidl.Base64.Benchmarks\bin\Release\netcoreapp3.0\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149.notcs:line 11144
   at BenchmarkDotNet.Autogenerated.Runnable_17.Run(IHost host, String benchmarkName) in C:\dev\git\Base64\perf\gfoidl.Base64.Benchmarks\bin\Release\netcoreapp3.0\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149.notcs:line 11088
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\dev\git\Base64\perf\gfoidl.Base64.Benchmarks\bin\Release\netcoreapp3.0\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149\57b5b4f8-0ef5-4b3d-9dc8-7f89889b3149.notcs:line 77

The donet SDK is in version 3.0.100-preview-009791.

Am I missing something?

ArrayPool-data should be cleared to be safe

E.g.

Base64/source/gfoidl.Base64/Internal/Base64Encoder/Base64Encoder.Encode.netstandard2.0.cs

Line 39 in 0d254ae

ArrayPool<char>.Shared.Return(arrayToReturnToPool);

Base64Url throws MalformedInput when status InvalidData is expected

The api are thought to be exception-less, that why they use OperationStatus.
Base64.Url can throw a FormatException (malformed input) in some cases. Maybe it's better to not throw the exception and just return OperationStatus.InvalidData?

	public static void WriteVector256(this ref byte dest, Vector256<sbyte> vec)
	{
	// As has better CQ than WriteUnaligned
	// https://github.com/dotnet/coreclr/issues/21132
	Unsafe.As<byte, Vector256<sbyte>>(ref dest) = vec;
	}

	private static readonly Base64Encoder s_default = new Base64Encoder();
	private static readonly Base64UrlEncoder s_url = new Base64UrlEncoder();

	if (end == EncodingPad) padding++;
	if (Unsafe.Subtract(ref end, 1) == EncodingPad) padding++;

	// Threshoulds found by testing -- may not be ideal on all targets

	if (data.Length < 26)
	return Convert.ToBase64String(data);

	Vector256<sbyte> t0 = Avx2.Permute2x128(c0, c1, 0x20);
	Vector256<sbyte> t1 = Avx2.Permute2x128(c0, c1, 0x31);

gfoidl / base64 Goto Github PK

base64's People

Contributors

Stargazers

Watchers

Forkers

base64's Issues

Comparison of AVX2 to SSSE3

Hardware info

Recommend Projects

Recommend Topics

Recommend Org