Comments (4)
Thank you for your interest in our work!
For reader data, we used the official HotpotQA Wikipedia corpus, which contains all of the first paragraphs of a 2017 Wikipedia dump. You can also download the corpus from HotpotQA official website https://nlp.stanford.edu/projects/hotpotqa/enwiki-20171001-pages-meta-current-withlinks-abstracts.tar.bz2 or other previous works repo, e.g. https://github.com/facebookresearch/multihop_dense_retrieval. With some data reformatting, you should be able to get the same content as in hotpot_corpus.jsonl.
from udt-qa.
I really appreciate for your response. I have two more question.
In the evaluation of FiE_reader using the provided hotpot_dev_reader_2hops.json, an EM score of 68.2 was obtained, as mentioned in the paper. However, when I conducted the evaluation from start to finish, I obtained an EM score of 61.9 which has big gap with 68.2.
The model I used is cos_nq_ott_hotpot_finetuned_6_experts.ckpt, and the data consists of hotpot_wiki_linker* at the span stage, hotpot_wiki_retriever* at the linking stage, and hotpot_corpus.jsonl at the cos stage (In cos stage, single retrieve, rerank, expanded retrieve, link from hop1 passage, and rerank was executed). After the cos stage, I fed top1 path into the reader. During the execution of train_qa_hotpot.py, hotpot_reader_checkpoint_best.pt was used both of the case.
Iām wondering if I ask you about the possible reasons for these discrepant results.(I am curious whether a different model other than cos_nq_ott_hotpot_finetuned_6_experts.ckpt was used to create hotpot_dev_reader_2hops.json.)
Also, I am curious about the difference between hotpot_wiki_linker* and hotpot_wiki_retriever*, both of which are computed hotpot corpus embeddings.
Thank you.
from udt-qa.
Just to clarify, "The model I used is cos_nq_ott_hotpot_finetuned_6_experts.ckpt" this is correct, we do not need any embeddings at span stage, and we use "hotpot_wiki_linker*" at linking stage.
After the cos stage, we actually run the https://msrdeeplearning.blob.core.windows.net/udq-qa/COS/models/hotpot_path_reranker_checkpoint_best.pt to get a top1 path from top100 cos results, and then that top1 is fed to hotpot_reader_checkpoint_best.pt. (This is also explained in Appendix B and Appendix C)
hotpot_wiki_linker* are computed with expert id 3 and hotpot_wiki_retriever* are computed with expert id 1, as shown in our Table 7, running inference with different experts actually leads to quite different results. So it's important to use the correct experts/embeddings at every stage.
from udt-qa.
Thank you for your kind assistance. My curiosity has been resolved thanks to you :)
from udt-qa.
Related Issues (8)
- Unable to Download Required Resources for Reader Evaluation HOT 7
- Public access is not permitted on this storage account.. HOT 9
- what is the ott_expanded_train_iter2 dataset in COS training script HOT 2
- Access Issue with Data Download from Storage Account HOT 4
- Questions Regarding UDT-QA Knowledge Base HOT 1
- Missing Files for Running COS Inference on OTT-QA HOT 5
- migrat UDT-QA data and models to HF HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from udt-qa.