This was going to be an unofficial implementation of file based pipeline primitives.

However, as I learnt more about pipelines and the code that exists already I stopped to benchmark my ideas. As a result of the benchmarks, detailed below, I've decided there is no need for dedicated file-based pipelines because stream adapters are already efficient enough.

Thus, this repo is now a benchmarking report rather than a library.

UPDATE Better code/benchmarks as suggested by @feO2x in #1 suggest no changes to recommendations.

DISCLAIMER: THIS CODE IS EXPERIMENTAL, I AM NOT AN EXPERT, THERE MAY BE MISTAKES

(Full results and conclusion at bottom of this document)

• The native pipeline file reader I tested was very memory efficient but also very slow
  • Results only calculated for Windows
  • Native code is tricky to get right - there are lots of gotchas
    • Windows doesn't even do async reads until the buffer gets to 64000 bytes
• When FileOptions.SequentialScan | FileOptions.Asynchronous are set performance universally gets worse (except for the native pipeline for which the option does not apply)
• FileStream (or stream) pipeline adapters (like Nerdbank.Streams):
  • are recommended if you're concerned about memory usage
  • Are slower than the FileStream sync and async APIs
• All pipeline adapters and async methods are slower than plain old FileStream sync reads
  • This is true whether or not FileOptions.Asynchronous is set
  • Though if you need non-blocking I/O then straight read performance is arguably not your goal
• Use a large buffer size (e.g. 65536 bytes or greater) for best reading performance

You can read more about file performance for ordinary reading scenarios:

• In @feO2x's InsightsOnFiles
• Add at the .NET performance repo dotnet/performance#402

And look out for the stream adapters in .NET https://github.com/dotnet/corefx/issues/27246.

System.IO.Pipelines is a new IO concept introduced in .NET Core 2.1.

Pipelines to do several important things that were traditionally tricky:

• buffer management
• backlog control
• thread managment
• and suport for back pressure

Fore more information see these blog posts:

• System.IO.Pipelines: High performance IO in .NET
• Pipelines - a guided tour of the new IO API in .NET, part 1

The pipeline APIs were released without any connectors. From Marc Gravell's blog post:

Here we need a bit of caveat and disclaimer: the pipelines released in .NET Core 2.1 do not include any endpoint implementations. Meaning: the Pipe machinery is there, but nothing is shipped inside the box that actually connects pipes with any other existing systems - like shipping the abstract Stream base-type, but without shipping FileStream, NetworkStream, etc.

Marc thus introduced Pipelines.Sockets.Unofficial to use in their .NET Redis library.

Since I needed to process large amounts of file-based I/O I decided to see if I could create my own file-based pipeline connector. Enter Pipelines.File.Unoffcial.

When the pipelines library was being developed, David Fowler created a native Win32 implementation of a file PipeReader.

• Initial implementation: https://github.com/dotnet/corefxlab/commit/df543b2c8ef193a9e04f93cd424b91c821aecb99#diff-54667d4e0d8ffdddf979ebbfe5681938
• Last commit: https://github.com/dotnet/corefxlab/commit/3bd0ffdef64b4510b408200cf4281d24b5742abd#diff-54667d4e0d8ffdddf979ebbfe5681938
• Removal: dotnet/corefxlab#2244, https://github.com/dotnet/corefxlab/pull/2244/commits/abba6a7e8faf530e5e2c8216d30ec01c73594e8c

As per the commits above it was an experiment that was abondoned. It would have been very hard to implement native file access for each platform.

You can find a modified implementation in ./src/Pipelines.File.Unofficial/FileReader.cs

I created my own naive implementation of a FileStream adapter which rather than bypassing FileStream (as done in David Fowler's implentation) uses the standard stream API.

You can find the implementation in ./src/Pipelines.File.Unofficial/FileStreamReader.cs

Realising I was perilously close to inventing some wheels, I added two previous (and generic) pipeline-stream adapters to the benchmarks:

• Sockets.Unofficial.StreamConnection by Marc Gravell
• Nerdbank.Streams.PipeExtensions by Andrew Arnott

These implementations are in found in their respective nuget packages.

All results were compiled with BenchmarkDotNet.

The benchmarking code can be found in ./tests/Pipelines.File.Unofficial.Benchmarks and the full benchmark report has been committed.

BenchmarkDotNet=v0.11.4, OS=Windows 10.0.17763.379 (1809/October2018Update/Redstone5)
Intel Core i7-8550U CPU 1.80GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview-009812
  [Host]     : .NET Core 3.0.0-preview-27122-01 (CoreCLR 4.6.27121.03, CoreFX 4.7.18.57103), 64bit RyuJIT
  DefaultJob : .NET Core 3.0.0-preview-27122-01 (CoreCLR 4.6.27121.03, CoreFX 4.7.18.57103), 64bit RyuJIT

• FileStreamSync: traditional FileStream sync reading of bytes
• FileStreamAsync: traditional FileStream async reading of bytes
• PipelineNative: This is David Fowl's Win32 Native file pipeline implementation
• PipelineAdapter: My own minimal FileStream adapter
• PipelineAdapter2: This is the pipeline stream adapter from Marc Gravell's Pipelines.Sockets.Unofficial
• PipelineAdapter3: This is the pipeline stream adapter from Andrew Arnott's Nerdbank.Streams

• File: one of two files, both containing incrementing little-endian ulongs, 4 MiB and 128 MiB, respectively
• BufferSize: the amount requested to read each iteration, in bytes. The NT API doesn't even do asynchronous reads until the buffer is at least 64k
• FileOptionsString:
  • One of two options either
    1. Async|Sequential which maps to FileOptions.SequentialScan | FileOptions.Asynchronous
    2. Or Sequential which maps to FileOptions.SequentialScan
  • Note: the FileOptions parameter has no effect on the PipelineNative method

Benchmarks with issues: ReadingBenchmarks.PipelineNative: DefaultJob [BufferSize=262144, FileOptionsString=Sequential, File=_int64_4.bin]

Note: the FileOptions parameter has no effect on the PipelineNative method

• In every single case specifying the additional FileOptions.Asynchronous flag resulted in slower performance than not
  • Where the Buffer, File, and Method are constant
  • Except in the PipelineNative method where this parameter is ignored
  • On average specifying FileOptions.Asynchronous was 3.56 times slower than not
    • In the worst case it was 9.22 times slower
    • In the best case it was 1.45 times slower (File=_int64_128.bin, Buffer=262144, Method=PipelineAdapter2)
• FileStream's sync methods are always fastest
  • For just FileOptions.SequentialScan | FileOptions.Asynchronous
    • PipelineNative performs better (remember FileOptions has no effect)
• FileStream's async methods are always faster than all pipeline methods
  • Except when FileOptions.SequentialScan | FileOptions.Asynchronous is set
    • PipelineNative performs better (remember FileOptions has no effect)
• BufferSize has a large affect on performance, decreasing read times as buffer size increases
  • this effect held for all groups (when Method, FileOptions, and FileSize are the same)
• The FileStream async method is consistently slower than its sync counterpart
  • Though the ratio shrinks from 1.92 times slower at small buffers sizes to only 1.11 times at large buffers
  • This effect is less severe when FileOptions.SequentialScan | FileOptions.Asynchronous is set
• For the native pipeline wrapper
  • If we split the results into three groups:
    1. FileOptions.SequentialScan: Results within groups are faster than PipelineNative
    2. PipelineNative sits in the middle
    3. FileOptions.SequentialScan | FileOptions.Asynchronous: Results with groups are slower than PipelineNative
  • The native API pipeline performs very poorly (up to 7 times slower than sync) at small buffer sizes when FileOptions.SequentialScan is set (Group 1 vs 2)
  • The native API pipeline performs very well (0.892 times sync) when FileOptions.SequentialScan | FileOptions.Asynchronous (Group 3 vs 2)
• Pipeline adapters (summarised for FileOptions.SequentialScan only)
  • The adapters also perform much better than the native pipeline at lower buffer
    • On average 2.582 time slower than FileStreamSync
    • But faster than PipeLineNative which was on average 5.36 times slower the FileStreamSync
  • All of the pipeline adapters have fairly consistent performance compared to each other
  • They also experience less variance across experiemnts
    • the slowest ratio is 3.19 times FileStreamSync and fastest is 1.76 times FileStreamSync (1.43 ratio variance over tests)
    • this is much less variance than experienced by the native pipeline (5.33 ratio variance over tests)
• File size differences:
  • the smaller file regularly is slower per MB than the larger file
    • Mean time taken scaled relative to file size
    • this effect is not seen with the PipelineAdapters (close to equivalent time taken per MB) at all buffer sizes
    • For PipeLineNative
      • this effect is prominent with buffer size 2048
      • and is non-existent at larger buffer sizes
• Memory:
  • The traditional file stream methods always allocate at least as much memory as the buffer
  • The FileStreamSync method with the FileOptions.SequentialScan | FileOptions.Asynchronous option allocates very large amounts of memory and should be avoided!
    • 20MB per operation in the worse case!
  • All pipeline methods allocate less memory than non-pipeline methods
  • The native pipeline never requires garbage collections for Gen 0, 1, or 2!
  • Only the traditional file stream methods have objects that survive to Gen 1 and 2, and only at buffer size 262144
  • The pipeline adapters do require garbage collection for Gen 0 objects:
    • At the smallest buffer size (2048), 4000 collections occurred!
    • But after buffer size 16384, the number of collections are equal or less than other methods, and sometime require no collections also

Anthony Truskinger (@atruskie)