MRR score uses a "mean" instead of "min" about spotlight HOT 6 OPEN

It's not necessarily incorrect, but I can see why you might want to do it differently.

Could you make a PR that keeps the current code as default, but allows the alternative as an option? You could pass a mode argument that's either mean or min.

from spotlight.

elanmart, I think worst case you can make a fix on repository you forked and share it here, I would also love to see it

from spotlight.

Hi @elanmart, are you still interested in contributing this option?

from spotlight.

Oh, yes, sorry. I'll submit the PR ASAP.

from spotlight.

Can I bump this. I really think that all literature discusses reciprocal rank as a per-user measure (per-query for information retrieval), and the average is taken across users. For instance, in the Croft book page 319, "reciprocal rank... is defined as the reciprocal of the rank at which the first relevant document is retrieved"

Therefore .min() should be the default.

Taking the .mean() is more similar of the reciprocal ranks is, in an adhoc way, to Mean Average Precision, and should be deprecated in favour of that.

Happy to contribute a PR. My own implementation also supports rank cutoffs and not dropping users with no relevant test items, but setting them to 0 in the array.

from spotlight.

bump

from spotlight.