Counter-intuitive result: more RAM = slower indexing (standard inverted indexes) about anserini HOT 3 CLOSED

My guess is that it's not actually faster, it's just taking a bit of work off indexing threads, and adding more work to merging, which is running asynchronously in its own threads.

Indexing boils down to updating large hash tables (inverted indexes) or graphs (HNSW). And the bigger they get, the slower the updates because you get more cache misses, etc.. So flushing N segments of size N is more costly than flushing N*2 segments of size S/2. But in-turn, this adds more work for merging. In your case, I'm assuming that you are not maxing out your CPU, so merging can take all the CPU it wants and indexing appears to be faster. But if you were trying to max out indexing so that indexing and merging would be competing for the same resources, then you would see a slowdown when decreasing the RAM buffer. Likewise if you told Lucene to run merging in indexing threads rather than their own threads (SerialMergeScheduler instead of ConcurrentMergeScheduler).

from anserini.

Ah, makes sense! I am using ConcurrentMergeScheduler.

Also, I guess that merging is (typically) disk throughput bound... and quite efficient since merging sorted lists is a linear time operation.

from anserini.

Right. It's rather efficient, but almost always still more expensive than doing less merging by accumulating bigger flush segments in the first place by configuring a bigger RAM buffer.

from anserini.