johngiorgi / bartscore Goto Github PK

Original BARTScore code skips documents for which all systems achieve the same automatic metric score

This is likely because, if all automatic metric scores are the same, their variance is 0, and correlation metrics like Spearman's R are undefined when variance is 0. This is probably harmless b/c it seems rare, but, in the most extreme case it happens for 10% of instances in REALSumm when computing rouge1_p_src_hypo. Our option are:

1. Leave it as is
2. Use an implementation of these correlation metrics that can handle this case (see here).