Files on Windows vfs lost when file names contain CJK punctuations about rclone HOT 6 OPEN

After some test, I find the bare minimum setup to reproduce this bug is

md Test
rclone.exe --local-encoding None mount --vfs-cache-mode=writes ./Test *

from rclone.

Does the file foo：bar.txt arrive on the destination OK?

Apparently it is uploaded correctly according to the log

2024/04/13 11:05:58 DEBUG : foo：bar.txt: vfs cache: starting upload
2024/04/13 11:05:58 INFO  : foo：bar.txt: vfs cache: upload succeeded try #1

Is it just that this file is no longer visible after it is uploaded?

from rclone.

Does the file foo：bar.txt arrive on the destination OK?

Apparently it is uploaded correctly according to the log

2024/04/13 11:05:58 DEBUG : foo：bar.txt: vfs cache: starting upload
2024/04/13 11:05:58 INFO  : foo：bar.txt: vfs cache: upload succeeded try #1

Is it just that this file is no longer visible after it is uploaded?

No, I've checked on the Cloudflare web console, there is no foo：bar.txt in the bucket.
And it doesn't matter if you use a remote backend or a local backend.
Even if you mount a local folder as a vfs, it doesn't write the file as well.

from rclone.

If I look in the VFS cache on windows I see this

 Directory of C:\Users\Dev\AppData\Local\rclone\vfs\local{sb0-v}\？\Z：\go\src\github.com\rclone\rclone

15/04/2024  16:43    <DIR>          .
15/04/2024  16:43    <DIR>          ..
15/04/2024  16:43                 8 foo‛：bar.txt
               1 File(s)              8 bytes
               2 Dir(s)   4,666,073,088 bytes free

And I see this in the log

2024/04/15 16:43:49 ERROR : foo：bar.txt: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to find cache file: CreateFile \\?\C:\Users\Dev\AppData\Local\rclone\vfs\local{sb0-v}\?\Z:\go\src\github.com\rclone\rclone\foo：bar.txt: The filename, directory name, or volume label syntax is incorrect.
2024/04/15 16:43:59 ERROR : foo：bar.txt: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to find cache file: CreateFile \\?\C:\Users\Dev\AppData\Local\rclone\vfs\local{sb0-v}\?\Z:\go\src\github.com\rclone\rclone\foo：bar.txt: The filename, directory name, or volume label syntax is incorrect.

So

• the file is still safe in the VFS cache
• it has a funny name with a ‛ before the ： despite encoding being none.
• the --local-encoding none is also affecting the VFS cache which it maybe shouldn't be.

I think that makes this issue related to

• #7456
• #6098

At minimum we need to make encoding none disable all encodings which it clearly isn't at the moment.

from rclone.

There are actually two problems.

The first is the same as #7456 and #6098: None encoding does not disable encoding.
It can be mitigated with the trick mentioned in #7456 .
And I have written a patch which I will open a pr later.

But it will still fail.
The problem is when creating temp file, it will use OS encoding (hardcoded), but try to decode it with encoding for the local filesystem (--local-encoding) when copying it to remote.
(technical detail: toOSPath for creating and localPath (inside local backend) for copy)
I have no idea how to unify them.

from rclone.

I have opened a pr that should resolve the original problem.
But I realized the "bare minimum setup" actually reveals the third problem:
The case when there is some special characters in relativeDirOSPath, the name of the cache directory.
But usually it is the name of the backend, which will not have some special characters.
In the "bare minimum setup", it is something like local{1pYER}\？\C：\xxx\yyy\zzz

from rclone.