Evaluation on DocRED about rebel HOT 2 CLOSED

Thx for your quick reply! btw I believe that JEREX's author only count correct for those entities whose all mention are correct :)

from rebel.

Hi there, I am not sure what you mean by "every entity must strictly consist of all gold mentions", is it whether REBEL outputs a triplet for each possible mention? If so, that is not the case, so I guess it would follow the (mention_head, relation, mention_tail) approach, we only used the first mention to an entity as the surfaceform that needs to be extracted by REBEL.

This is why REBEL is not evaluated on entity recognition, as it only extracts one mention for a given entity, and only mentions present in a triplet would be extracted. In JEREX (https://arxiv.org/pdf/2102.05980.pdf) for instance if I am not mistaken, they use the representation of all mentions in order to classify the relation. At the end, in both systems you obtain a prediction between entities, however here JEREX is also extracting all entity mentions, which may be what you are looking for. And now that you bring this question up, I wonder whether in JEREX they only consider a relation correct if all mentions are extracted. In their paper:

While prior joint models focus on mention-level
relations (e.g. Gupta et al. 2016; Bekoulis et al.
2018; Chi et al. 2019), we extend the strict evalu-
ation setting to entity level: A mention is counted
as correct if its span matches a ground truth men-
tion span. An entity cluster is considered correct
if it matches the ground truth cluster exactly and
the corresponding mention spans are correct. Like-
wise, an entity is considered correct if the cluster
as well as the entity type matches a ground truth
entity. Lastly, we count a relation as correct if its
argument entities as well as the relation type are
correct.

I always assumed that by entity arguments they meant just that they are the correct entities as head and tail. If they only count as correct those for which all mentions are correct, I believe the comparison would be a bit unfair in REBEL's favor. I will try to ask the authors of JEREX.

So thanks for bringing this up and I hope it answered your question :)

from rebel.