Is the MRR calculation reasonable now? about codesearchnet HOT 2 CLOSED

Hi @fengzhangyin

CodeSearchNet has two tasks:

a) Given a documentation comment (e.g. Python docstring) try to find the original code snippet that matches that comment. For this task, there is plenty of data and thus supervised machine learning methods can be used to train models.

• During training, we randomize the batch elements and ask the model to learn to pick the correct code from within the batch.
• During validation/testing, the batch elements are fixed (given that the evaluation dataset doesn't change) and therefore comparing among models is possible.
• We train our models using the MRR objective using a fixed validation ordering and batch size.

b) However, documentation comments are not necessarily representative of real code search queries. For this reason we have collected a small dataset of human-collected annotations (which is hidden behind the leaderboard submission).

• Relevance annotations are commonly evaluated using NDCG. MRR does not apply.
• The ability to rank the correct snippet highly, does not necessarily correlate with NDCG/human-relevance annotations since documentation comments differ from search queries.

How should you train the model? That's up to you. We suggest using MRR and the docstring<->code task, but feel free to pick any alternative you think it would work best. How do you pick the best model? Again it's up to you.

Have a look at the Technical Report for more info.

from codesearchnet.

Hi @mallamanis
Thank you very much. The explanation is very clear.

from codesearchnet.