amirveyseh / aaai-21-sdu-shared-task-1-ai Goto Github PK

Using the provided character_match.py script, I created a baseline system and evaluated it with the scorer script. The performance of the baseline does not seem to be correctly evaluated with the latter script, as I am getting the following results on both train and dev predictions:

Shorts: P: 0.00%, R: 0.00%, F1: 0.00%
Longs: P: 0.00%, R: 0.00%, F1: 0.00%
micro scores: P: 0.00%, R: 0.00%, F1: 0.00%
macro scores: P: 0.00%, R: 0.00%, F1: 0.00%
Official Scores:
P: 0.00%, R: 0.00%, F1: 0.00%

Evaluating your provided predictions.json file also gives the following:

Shorts: P: 100.00%, R: 100.00%, F1: 100.00%
Longs: P: 100.00%, R: 100.00%, F1: 100.00%
micro scores: P: 100.00%, R: 100.00%, F1: 100.00%
macro scores: P: 100.00%, R: 100.00%, F1: 100.00%
Official Scores:
P: 100.00%, R: 100.00%, F1: 100.00%

In the description of the task, you report Precision: 93.22%, Recall: 78.90%, F1: 85.46%. Are those regarding the test sets? Is it normal that the baseline script results 0% or 100%?

Wondering if you have the time stamps for your raw documents (i.e., the years in which each document was written).

Thanks!

Hi,
I was unable to locate the predictions file for the test data. Can I please get access to it?
Thanks