How to evaluate the supporting facts in the HotPotQA experiment? about learning_to_retrieve_reasoning_paths HOT 5 CLOSED

Hi,

thanks for your interest in our paper.

The HotpotQA dataset is a well-designed dataset, where the corresponding Wikipedia dump is provided along with the dataset, and all the passages' sentence boundaries are also given.
In the supporting fact extraction, what we need to output is something like this:

[article_title_1, [sentence_index_1, sentence_index_2, ...]], ...

so that we can compare the output with the ground truth, based on the key (article title) and its values (sentence indices).

Thanks

from learning_to_retrieve_reasoning_paths.

Thank you for your reply！I have a second thought

from learning_to_retrieve_reasoning_paths.

I noticed that in the process of Reasoning Path Retriever, node C is not output as a hidden state. Can you explain me?

I saw an explanation of your in the article: "After adding negative sampling to the training sample, you can train the Retriever separately (the negative sampling sample uses Reasoning Path that seems to be related to the problem but does not contain the answer to replace groundtruth, the purpose is to help the model Distinguish irrelevant paragraphs)”, then in this Retriever stage, there is actually no loss function. How do you judge that the node C is a Reasoning Path that does not contain an answer?

Thank you for your reply！！！

from learning_to_retrieve_reasoning_paths.

The above question is based on the model diagram. In the RNN part of the model diagram, there is no C node as the hidden state output.

from learning_to_retrieve_reasoning_paths.

Thank you for pointing it out! We've updated our paper on openreview, but haven't updated it on Arxiv as we've faced the maximum file size limitation (100 MB). I hadn't had a chance to fix them due to other projects but will try to update the Arxiv version to make the figures consistent.

from learning_to_retrieve_reasoning_paths.