Space reservation about securefs HOT 13 CLOSED

Here is the binary.

securefs-win.zip

from securefs.

Dear @billziss-gh, this issue may also be of interest for you.

from securefs.

• EncFSMP, which depends on Pismo File Mount driver, seems to be affected by the same issue
• CPPCryptFS (a Windows port of GoCryptFS), which depends on Dokany driver, works w/o delays

from securefs.

What is the underlying filesystem? NTFS or FAT32?

from securefs.

NTFS

from securefs.

@sergeevabc I am on vacation and do not have any means to check things more thoroughly.

@netheril96 is the best person to answer this. However it may be that securefs actually extends files by writing encrypted zeroes in the extended portion of the file. So files end up being written twice. First during a truncate call and then during the actual write.

from securefs.

The issue results from the lack of sparse files on Windows. securefs does not encrypt and write zeros when resizing the file. It writes only two blocks, one at the old end, and one at the new end. But Windows fill the intermediate with zeros, and that is what takes a long time.

I have no idea now how to work around it.

from securefs.

@netheril96, why do you claim there are no sparse files on Windows, are you sure?
E.g. Tixati, a cross-platform torrent client, uses sparse method for files allocation. Works smoothly.

from securefs.

@netheril96

The issue results from the lack of sparse files on Windows. securefs does not encrypt and write zeros when resizing the file.

My understanding of sparse files on NTFS is that they are not enabled by default. One must explicitly enable them with FSCTL_SET_SPARSE.

It writes only two blocks, one at the old end, and one at the new end. But Windows fill the intermediate with zeros, and that is what takes a long time.

You could enable sparseness on NTFS and "punch a hole" using FSCTL_SET_ZERO_DATA . Given that yours is a portable file system and that NTFS sparse files may have tradeoffs that we do not currently understand, I do not know that I would necessarily recommend this approach.

Is it not possible to somehow eliminate that last encryption block?

Many file systems avoid the need to write zeroes upon file extension (truncate) by maintaining both a filesize and a validsize. The filesize maintains the size of the file as seen by user processes. The validsize contains the size of valid data within the file. It is always validsize <= filesize.

Here is how the validsize optimization works. Suppose that at some time it is validsize==filesize==S. Then the file system receives a truncate(T) with S < T. In this case the file system simply updates the filesize attribute: validsize==S < filesize==T. If it later receives a read referencing the range [S, T) it notices that the range is beyond validsize and simply returns zeroes (0) for that range. The validsize attribute is only updated when receiving a write beyond the existing validsize.

NTFS itself implements this optimization even without sparse files.

from securefs.

I added a workaround in the latest commit. I'm not sure if it preserves the semantics on Windows though, especially if it syncs with Unix filesystems via cloud. To be more specific, when I extend the file size with SetFilePointerEx, and then read beyond the physical limit, will the data returned be all zeros? If not, then some issues may arise out of this change.

from securefs.

@netheril96

To be more specific, when I extend the file size with SetFilePointerEx, and then read beyond the physical limit, will the data returned be all zeros?

Yes, if you use SetFilePointerEx to move the file pointer past the end of the file and then do a SetEndOfFile or WriteFile the file system will guarantee that the range between the previous end of file and the start of the SetEndOfFile or WriteFile will be zeroes (0). NTFS will do this using the validsize trick (when using SetEndOfFile) or by filling the "hole" with zeroes (0) (when using WriteFile with a non-sparse file) or by possibly recording the "hole" in metadata and possibly filling some parts of it with zeroes (when using WriteFile with a sparse file).

In any case a subsequent ReadFile in the extended range is guaranteed to return zeroes (0).

from securefs.

Then commit 351cb13 should have fixed it.

from securefs.

Ahem, this issue is pretty serious to close it as resolved without verification step, isn’t it?
Being a mere user without compiler, me and possibly others are not able to test it, so where’s binary?

from securefs.