mmap input files rather than reading about n2 HOT 5 OPEN

Relevant code in Android's Ninja fork. (Not sure how to get a permalink to that commit but the file is unlikely to change much in the future, but it's the function MapFile.)

from n2.

I wrote a mini benchmark here.

On this Mac Air M2 for a random 1.4gb file:

without madvise:

  ./target/release/read-file mmap $F ran
    1.25 ± 0.01 times faster than ./target/release/read-file read $F

with madvise:

  ./target/release/read-file mmap $F ran
    1.36 ± 0.02 times faster than ./target/release/read-file read $F

So I conclude at least for this hw/OS that mmap is faster, and madvise helps too.

from n2.

Relevant code in Android's Ninja fork. (Not sure how to get a permalink to that commit but the file is unlikely to change much in the future, but it's the function MapFile.)

Sorry, this link should be permanent.

from n2.

Just a thought regarding "correctness" more than performances.

Is there any risk of the mmaped build.ninja changing during the build?
I had a bad experience trying to mmap user-controlled file, an update of the file at the wrong time could cause my process to crash with SIGBUS.
Edit: it could just be I didn't use the proper type of memory mapping.

The build.ninja file can be regenerated during build by the build system (I think cmake does this when a CMakeLists.txt changes), can this invalidate the memory mapping?

from n2.

can this invalidate the memory mapping?

The Linux man pages says it's undefined behavior. The Mac ones don't say either way. It is true that we wouldn't want to preserve the mapping beyond the point where the file might be rewritten though.

from n2.