LRU cache preventing unmapping of ROOT files about unroot.jl HOT 7 CLOSED

Nice finding, I think basically the memorized function shouldn't take ROOTFile as input argument because that makes the key lingers in LRU cache.

But this shouldn't be a problem other than producing bigger mapping number right? Since we only MMap the file, it's technically not real memory used

from unroot.jl.

Moelf wrote: "But this shouldn't be a problem other than producing bigger mapping number right? Since we only MMap the file, it's technically not real memory used."

It uses virtual memory address space. This address space is quite large on 64-bit architecture and much more limited on 32-bit one. It's any case a good practice to release resource, when no more needed.

I've also observed that it increases the resident set size (RSS). I expect it's an artifact, because in my test it went well beyond the physical memory with no impact on performance. Nevertheless, it can be worrying for the user and can be a real problem if a limit is set on RSS usage (see ulimit). My linux kernel version is 5.10.0.

My guess is that file data loaded by the kernel in the memory buffer are counted, even they have been discarded and until the memory is unmapped.

Philippe.

from unroot.jl.

Thanks for the investigation and fix proposal Philippe!

I have not have time to reproduce the issue yet but was wondering if closing the file properly (which is also a good practice ;)) in your example would make any difference.

from unroot.jl.

It doesn't change. You need the ROOTFile instance and its fobj to be garbage collected, which was prevented by the reference in the LRU cache.

from unroot.jl.

Yes, you are right!

I think #261 should work, but have not verified yet.

from unroot.jl.

That PR also has tests

from unroot.jl.

Yep, saw them already. We can merge if you agree :)

from unroot.jl.