Allow trec_eval to take symbols representing standard qrels (instead of full qrel files) about anserini HOT 7 CLOSED

@lintool What should happen in the case where we have two file names that would in theory have the same associated symbol?

Ex. Symbol msmarco-v2-passage could be associated with qrels.msmarco-v2-passage.dev.txt and qrels.msmarco-v2-passage.dev2.txt.

Would it be okay to simply require symbol names to be longer?

In your example, msmarco-v2-passage would not be a valid symbol. User would need to specify either dev or dev2.

As long as the file names are not identical (and they shouldn't be), it's fine. We specify which qrels we're using in the command line to trec_eval.

from anserini.

working on this

from anserini.

@xpbowler any progress here?

from anserini.

Sorry! I've been busy with finals these past 2 weeks. Last exam is tomorrow, so I'll get back on this after.

from anserini.

@lintool What should happen in the case where we have two file names that would in theory have the same associated symbol?

Ex. Symbol msmarco-v2-passage could be associated with qrels.msmarco-v2-passage.dev.txt and qrels.msmarco-v2-passage.dev2.txt.

Would it be okay to simply require symbol names to be longer?

from anserini.

For the symbols, let's use the bindings here: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/eval/Qrels.java

So instead of

java -cp anserini-0.35.1-fatjar.jar trec_eval -c -M 10 -m recip_rank tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt run.msmarco-v1-passage.dev.bm25.txt

We can just do:

java -cp anserini-0.35.1-fatjar.jar trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset run.msmarco-v1-passage.dev.bm25.txt

from anserini.

hey @xpbowler just to prevent duplicate work, I think @DanielKohn1208 is on this!

from anserini.