Some doubts on the dataset tested about easyedit HOT 4 CLOSED

Thank you for your interest in EasyEdit. 😊

• To test the model's ability to take advantage of edited facts, we used gpt4 to generate a one-hop question about edited facts. So the editing performance was tested on the dataset zsre_mend_eval_portability_gpt4.
  • That is, we extend the original ZsRE dataset for more robust evaluation.
• Yes, the original ZsRE dataset could not be used for portability evaluation.

from easyedit.

OK, Thank you very much for your qucik response.

So the portability evaluation in the same performance table with original dataset is only on 1037 samples? I mean, maybe too small to be representative compared to the original amount of zsre? And the probability problem is only one per sample.

Dont mind it, it is a little comment :). Anyway it is a good work. Thanks.

from easyedit.

Thank you for your advice. 🤗

• In fact, 1037 samples is enough to measure the performance of different editing methods. For example, in the sequence editing scenario, the editing capacity of most editing methods is <=100.
• You're right. In our experiments, only one one-hop entity is considered for each editing case. We test the model's ability to apply edited facts, unabling to identify all of the affected one-hop link entities.

from easyedit.

If your problem has been resolved, please help close this issue.

from easyedit.