Incorrect F-1 implementation about donut HOT 2 CLOSED

BTW, if I use the provided code, I got F-1 91.5 on CORD dataset, which closely matches the numbers in the paper.

If I include the unmatched ground truth as false negatives, the F-1 is 83.6.

from donut.

Hi, thank you for reporting the issue. No, it is not intended. It seems that this issue has occurred during the recent refactoring to 1.0.6. donut-python>=1.0.7 will resolve the issue (pip install donut-python --upgrade).

Regarding the different scores, we have tested various versions/types of F1, and the current implementation in this repository is the most rigid one. For example, it counts the case where the key is exact-matched but the value is not (e.g., "menu.nm":"Cake" & "menu.nm":"cake") as both FP and FN. Note that this might be too severe for all methods in an end-to-end setting (regardless of the OCR dependency. Also, this is a reason for using a TED-based metric). For instance, if a single character is missed in the OCR, there is no chance for the conventional tagging-based models. For the various versions, the overall trends among the methods were the same in our analyses. The current version will return a score of around 84.

python test.py --dataset_name_or_path naver-clova-ix/cord-v2 --pretrained_model_name_or_path naver-clova-ix/donut-base-finetuned-cord-v2
100%|█████████████| 100/100 [00:35<00:00,  2.80it/s]
Total number of samples: 100, Tree Edit Distance (TED) based accuracy score: 0.9374973163596032, F1 accuracy score: 0.8406020841373987

The current script (donut-python>=1.0.7) is simple and easy to understand, and we further improved and fixed our script during the refactoring. With the latest evaluation script, we updated the scores at https://arxiv.org/abs/2111.15664. See https://github.com/clovaai/donut#test also. Hope this helps :)

from donut.