Querying the training set: runtime tradeoff for large k about pynndescent HOT 2 OPEN

Thanks so much for the reply and these suggestions! I'm not yet doing any of those so I will definitely try them out.

from pynndescent.

For such a large k, yes, there is going to be some potential benefit to building with a decently large k to ensure a quality index, and then querying. I can't say for certain what k would be good, but probably on the order of 80 to 160 would be a good start.

Some other options for dropping the runtime: you can max out the number of trees used at something smaller than it would pick given the size of the dataset. I suspect 16 trees are likely enough. That can save some time. Another trick would be to l2-normalize all your vectors and use "dot" as your distance metric instead of cosine (these will be equivalent). That will amortize the normalization cost into the pre-processing instead of being redone on every distance computation.

You may already be doing these of course.

Best of luck.

from pynndescent.