Why the results are the same? about odex HOT 1 CLOSED

Hi,

Thank you for your interest in our dataset!

First of all, the result in your screenshot comes from our ablation study (Appendix E.1.3) that does not remove trailing whitespaces, and it should not be referred to as the most standard evaluation results of the CodeGen model. Please check Table 4 in the main content instead! (and pass rate does increase w.r.t. k)

Second, we've run the experiment multiple times (>=5) and it gives pretty consistent results, i.e., that pass@k stays the same with larger k (>=2). We encourage you to run the experiment to verify. We are also happy to provide the exact prediction file for you to examine.

To give a brief description of why this happens: (1) for most of the examples, pass@k reports 0.0 or 1.0 for all k from 1 to 10, which means either all predictions are correct, or all predictions are incorrect, so that makes no changes in pass rate at different k values; (2) for a small number of cases, only 1 or 2 out of 10 are correct, so that result in the small increase from pass@1 to pass@2.

Hope this provides sufficient information to address your concern!

from odex.