chaitanyamalaviya / expertqa Goto Github PK

Hello authors, thank you for the great work!

Quick question regarding the released data data/r2_compiled_anon.jsonl: it seems many entries have URLs only for claim["evidence"], rather than URL+ passage.

I was wondering if this means the evidence is inaccessible and the claim thus falls into the support=="N/A" category. But this does not seem to be always true after further inspection.

Evidence passages are necessary for model evaluation (please correct me if I am wrong). Could you share the best way to retrieve them for these URLs? Thank you!

It seems that the r1_data.jsonl and r1_data_bingchat_balanced.jsonl files mentioned in collect_answers.sh do not appear in your project.

Hi ! I’m Quentin from HF :)

Thanks for sharing the dataset, I believe it will be used a lot to evaluate LLMs! Especially since factual correctness and attributions are imo at the heart of many challenges nowadays.

I was wondering if you planned to share the dataset on Hugging Face ? This way researchers can load it in one line of python, and there is also a nice dataset viewer on the website to visualize the data.

This problem occurred when I was running the code, and I don’t know how to solve it for the time being.

Thanks for your great job. When I try to git clone the repo with:

git lfs install
git clone https://github.com/chaitanyamalaviya/ExpertQA.git

an error occurs and it fails to download data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl:

Cloning into 'ExpertQA'...
remote: Enumerating objects: 125, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 125 (delta 13), reused 33 (delta 11), pack-reused 85
Receiving objects: 100% (125/125), 57.08 MiB | 3.63 MiB/s, done.
Resolving deltas: 100% (31/31), done.
Updating files: 100% (71/71), done.
Downloading data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl (298 MB)
Error downloading object: data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl (5e6afb0): Smudge error: Error downloading data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl (5e6afb0453bc8c5fabfe43fbef0b205ceb3175694640cfd1f5d47a97900d0ac7): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to '/Users/chengan/Desktop/tmp/ExpertQA/.git/lfs/logs/20240111T174938.90161.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

May you have any idea?

翻译成英语：

Is there any difference between the files data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores_anon.jsonl and data/r2_compiled_out_corrected_revised_atomic_w_evidences_factscores.jsonl, or are they essentially the same?