[QST] num_threads on ANN latency benchmark about raft HOT 2 OPEN

@pazzap123 your understanding of latency and throughput modes are mostly correct, except that both the latency and throughput mode can have a batch size > 1. As a result, we are in effect measuring the latency of each batch.

To make this comparison fair, we use a thread pool for the cpu-based algorithms, since the gpu-based algorithms will process a batch at a time by saturating the gpu.

from raft.

Thank you! I made a mistake by trying to use the command line -threads to control available threads for batching in the latency test even though the help says it is for the throughput test (I set -batch-size=10 and nthreads=1 and then nthreads=10, and assumed the batch would get serialized on case1 and all 10 would be sent together in case2). I have to use numThreads in the conf files instead for this.

When I compare the performance on a high core count system of
-> Throughput test with batch-size=10, -threads=10
-> Latency test with batch-size=10, numThreads=10

The throughput test performs better, i assume because of two effects:

1. All queries have to complete in a batch for the latency test before it tries another iteration, so the longest leg impacts that batch's time; For throughput, as soon as any query is done, another can be jammed in (meaning if query 1 of those 10 finished, another query can be sent in from the next iteration)?

2. Maybe a second overhead related to thread synchronization overhead (pthreads) at the boundaries of each batch since it has to wait for all to complete and synchronize.

I tried to illustrate it with a simple example of how things would work with some long running queries.

from raft.