Remove memory mapping of DTZ files. about tb HOT 7 OPEN

I think that if you memory map a file, only the parts that are accessed are effectively loaded into physical memory. So memory mapping already functions as a cache.

from tb.

https://en.wikipedia.org/wiki/Memory-mapped_file

from tb.

Also very likely if the same file is memory mapped by multiple processes, the different mappings share their physical memory.

from tb.

You are completely missing the point. Having the files mapped creates enormous problems, especially using 7 piece TBs. Arguing that mapping acts like a cache is a water is wet statement. The point is that mapping buys nothing for typical use cases, but causes extreme memory issues for zero benefit. There is zero reason to try to speed access to DTZs at the expense of RAM that could be used for more productive purposes. I'm running multiple engines using this code and it turns RAM into such a train wreck, I need constant reboots as the system (Windows) can't even recover after the engines are halted. (Converting Leela training data to Stockfish format). The same would be true running say 32 Stockfish's in a match using let say 6 piece TBs. Say I have enough RAM to memory map the WDLs but not the DTZs being probed. Say 8 Stockfish's are in a late stage game mapping DTZs to RAM when the others need to map WDL for use in search. The WDL access times will get clobbered. WDLs need speed not DTZs.

DTZs should NEVER be mapped.

from tb.

You seem to believe that if a file is memory mapped then the whole file is loaded into memory. This is simply not true. Only the pages that are accessed are loaded into (physical) memory. Moreover such pages can be ejected again if they are no longer used. Recall that DTZ files are only accessed at root. In other words very rarely.

from tb.

You seem to believe in the Easter Bunny. Where did I say the whole file is mapped please quote me. You've clearly never used this code as I described above and seem to have a lot of "theory." I have a lot of practice. I have described the problem adequately well, if you don't believe in it - having a religious debate about your "beliefs/theories" is not a good use of my time. I suppose in your beliefs, I "imagine" the problems this causes and imagine that windows will leave my used RAM stuck at nearly 100% for hours after these processes end, and I imagine the immense disk thrashing when other programs need RAM during engine use or after all the engines are closed. I must also imagine my page file being hammered like I have no RAM free just as the OS reports.

Please stop. I know I will, good bye.

from tb.

I think there is something wrong with your setup, or else you are misdiagnosing your issues (note that a modern OS will always try to use 100% of available physical memory).

An exited process will free its allocated memory, so if that doesn't happen then something is seriously wrong. Moreover memory used by a memory mapped file is by definition "file backed memory". So when reclaimed, it will not cause trashing (trashing is caused by data which is swapped out to disc when memory is tight).

from tb.