microsoft / copyonwrite Goto Github PK

Mac version is TODOs and NotImplementedExceptions

When running a Parallel.ForEach(10000, i => CloneFile("sourceFile", $"file{i}")) there are typically 3-5 of 10K calls that fail with a winerror 134. This might be a corruption issue. x64 test.

Is it possible/safe to create a clone of a locked file using shadow copy?

Need more testing and regression testing for 32-bit x86. When trying to integrate library into MSBuild net472 x86 the process core-dumps with memory corruption errors when running CloneFile().

MSBuild Copy task has recently added this important logic:
https://github.com/dotnet/msbuild/blob/c36a54ed3308d1516ffe1a86b9086c42e4ca996f/src/Tasks/Copy.cs#L288-L294

If the Copy task sees that the destination exists, it deletes it before overwriting. This is important, because if the destination is hardlinked to some other file, previously the Copy task used to effectively overwrite all other hardlinked copies of the content, and this caused corruption that's hard to investigate.

Let's ensure the Copy logic in this repo does the same thing (delete the file before overwriting to "unlink" it in case it's a hardlink)

Reported by a Windows customer - reading a container volume's metadata requires escalation that should not be necessary, resulting in an UnauthorizedAccessException.

F:\w\b\some.proj(13,5): error MSB4061: The "Copy" task could not be instantiated from "C:\Users\XXXX.nuget\pa ckages\microsoft.build.copyonwrite\1.0.265\Sdk..\build\netstandard2.0\Microsoft.Build.CopyOnWrite.dll". F:\w\b\some.proj(13,5): error MSB4061: System.TypeInitializationException: The type initializer for 'Microsoft.Build .Tasks.Copy' threw an exception. ---> System.UnauthorizedAccessException: Failed retrieving volume information for \?\Volume{629458e4-0000-0000-0000-0100000000 00}\ with winerror 5
F:\w\b\some.proj(13,5): error MSB4061: at Microsoft.CopyOnWrite.Windows.VolumeInfoCache.GetVolumeInfo(VolumePaths volumePaths) in D:\CoW\lib\Windows\VolumeInfoCache.cs:line 153 F:\w\b\some.proj(13,5): error MSB4061: at Microsoft.CopyOnWrite.Windows.VolumeInfoCache.BuildFromCurrentFilesyste m() in D:\CoW\lib\Windows\VolumeInfoCache.cs:line 36 F:\w\b\some.proj(13,5): error MSB4061: at Microsoft.CopyOnWrite.CopyOnWriteFilesystemFactory.Create() in D:\CoW\l ib\CopyOnWriteFilesystemFactory.cs:line 48

From unit tests - sporadic failures locally and in CI pipeline:

Assert.ThrowsException failed. No exception thrown. MaxCloneFileLinksExceededException exception was expected.

Implications:

• We have some sort of leak in the cloning logic that does not complete all clones before attempting to set the (limit + 1) clone.
• Windows itself has lazy accounting for the clone limit, i.e. it's a "loose" limit.

When attempting to copy onto an alternative data stream, a BSOD is caused.

Simple reproduction (F:\ is a Dev Drive):

File.WriteAllText($@"F:\New Text Document.txt", "a");
File.WriteAllText($@"F:\New Text Document 2.txt", "b");
File.WriteAllText($@"F:\New Text Document 2.txt:x", "c");
Microsoft.CopyOnWrite.CopyOnWriteFilesystemFactory.GetInstance().CloneFile($@"F:\New Text Document.txt", $@"F:\New Text Document 2.txt:x");

I discovered this while trying to implement dotnet/runtime#86681, can you please raise this internally so we can try to get it quickly fixed in Windows, after verifying it on your end. For now, I will add a workaround to skip for ADS and revert all of my testing changes on my local machine, so I can hopefully get a PR in a workable state soon.

Note, that my local version doesn't have the issue where it may get confused about which volume it's on by a path like this, it should definitely know it's on F:\ as it uses Windows APIs to correctly determine the volume. I'm happy to triple check this if you'd like, but a BSOD should obviously not be caused regardless.

Windows version:
Edition: Windows 11 Pro Insider Preview
Version: 22H2
OS build: 23481.1000
Experience: Windows Feature Experience Pack 1000.23481.1000.0

Image of BSOD if you want it for some reason (which I have because it's in a VM):

Windows-only.

Symptoms: When cloning a single source file in parallel (multiple threads or processes) to multiple destinations, sometimes success is returned but the file region assignment is never completed. This results in the destination file having all zeroes for its content.

Cause: When a file is not completely flushed to disk when the region clone operation starts, ReFS tries to flush the file to disk first. There is a race on multiple threads where failure to flush is ignored and the region clone proceeds anyway.

Workarounds:

• Serialize cloning system-wide per source path. This library takes this approach by default for single-process cloning by using an in-memory dictionary. You can opt into system-wide serialization using kernel mutexes by specifying useCrossProcessLocksWhereApplicable = true when calling CopyOnWriteFilesystemFactory.GetInstance().
• Ensure the source file is completely flushed to disk before cloning. This can be accomplished through one of the approaches below. Note that if you use these approaches, you can increase performance of cloning by using CloneFileFlags.NoSerializedCloning on your CloneFile calls.
  • Using the FlushFileBuffers API to force the file to be flushed from memory. This should be called at the end of writing the source file to disk while the file write handle is still open. Alternately it could be called on a new handle to the file opened with GENERIC_WRITE.
  • When writing the source file, open the file handle with FILE_FLAG_NO_BUFFERING. However, note this requires the code writing to the file to deal with writing chunks aligned with the sector size of the underlying volume, and using chunks that are a multiple of the sector size.
  • When writing the source file, open the file handle with FILE_FLAG_WRITE_THROUGH. This forces a flush on every write, which can decrease performance significantly.

Resolution: We currently have only workarounds (see above) and are awaiting resolution with the Windows team.

This issue tracks resolution in the Windows codebase and other approaches to work around the problem. Related to #1. PRs that have added workarounds:

• #6 - added single-process (in-memory dictionary) serialization of cloning of individual files.
• #9 - added optional cross-process serialization using kernel named mutexes, one per file path.
• #13 - documentation
• (See any linked future PRs)

If source file is NOT sparsed and CloneFlag.DestinationMustMatchSourceSparseness is set, block-cloned destination file should be NOT sparsed, however destination file is sparsed.

This Windows command can be used to check a file sparness:
fsutil sparse queryflag C:\Temp\myfile.vhdx

To set the destination file as not sparsed a FILE_SET_SPARSE_BUFFER structure should be used, currently a bool[] value is used.

This sample code successfully sets a file as not sparsed. Tested on Windows 2016 1607 and Windows 10 21H2. The only variation with the code on CloneFileAsync method is the use of a FILE_SET_SPARSE_BUFFER struct.

var FILE_SET_SPARSE_BUFFER = new FILE_SET_SPARSE_BUFFER
{
      SetSparse = false
};

if (!NativeMethods.DeviceIoControl(
    destFileHandle,
    NativeMethods.FSCTL_SET_SPARSE,
    FILE_SET_SPARSE_BUFFER,
    Marshal.SizeOf(FILE_SET_SPARSE_BUFFER),
    null,
    0,
    ref numBytesReturned,
    IntPtr.Zero))
 {
     int lastErr = Marshal.GetLastWin32Error();
     NativeMethods.ThrowSpecificIoException(lastErr,
     $"Failed to turn off file sparseness with winerror {lastErr} for destination file '{destination}'");
  }

Linux version is TODOs and NotImplementedExceptions

The ReFS related code is non-atomic. If the CoW code crashes, the real filename with incomplete content will remain. Consider use of a temp file extension like .cowtemp then rename after clone complete.

Sporadic failure. Occurs in either the 32 or 64 bit test suites. One interesting data point is that the test failure seems to occur on a test iteration that is a multiple of 510 (test510, test1020 seen in practice).

Test method Microsoft.CopyOnWrite.Tests.CopyOnWriteTests_Windows.ReFSPositiveDetectionAndCloneFileCorrectBehavior threw exception:
System.AggregateException: One or more errors occurred. (Assert.AreEqual failed. Expected:<1234abcd>. Actual:<\0\0\0\0\0\0\0\0>. B:\Stress1\test510) ---> Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.AreEqual failed. Expected:<1234abcd>. Actual:<\0\0\0\0\0\0\0\0>. B:\Stress1\test510

Stack Trace: 
<>c__DisplayClass10_1.b__0(Int32 i) line 241
<>c__DisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location --- <>c__DisplayClass19_01.b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
Replica.Execute()
--- End of inner exception stack trace ---
TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure) Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally)
--- End of stack trace from previous location ---
Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) Parallel.For(Int32 fromInclusive, Int32 toExclusive, Action1 body)
CopyOnWriteTests_Windows.StressTestCloning(String refsRoot) line 239
CopyOnWriteTests_Windows.ReFSPositiveDetectionAndCloneFileCorrectBehavior() line 157
CopyOnWriteTests_Windows.ReFSPositiveDetectionAndCloneFileCorrectBehavior() line 157
ThreadOperations.ExecuteWithAbortSafety(Action action)

Standard Output: 
Creating file with size 2147484672
Running parallel stress test for 8175 CoW links
Running parallel stress test for 8175 CoW links

I'm debugging Microsoft.Build.CopyOnWrite.dll and I'm not seeing SourceLink.