• Paper

  • Yewon(Chloe) Han (2023)

• Prior projects

  • factcheck-ko-2021
  • factcheck-ko-2020

git clone https://github.com/OnedayOneyeah/factcheck-ko-2023.git
cd factcheck-ko-2023
conda create -n factcheck-ko-2023 python=3.8
conda activate factcheck-ko-2023
pip install -r requirements.txt

Download the data for training here. Save the files in ./data/

• data/wiki_claims.json: Human-Annotated Dataset for the Factcheck
• data/train_val_test_ids.json: Lists of claim ids for train/validation/test split

• Documents are available here.

• Save SS checkpoint (download) in ss/checkpoints

• Save RTE checkpoint (download) in rte/checkpoints

• Save additional RTE checkpoints in new_rte/checkpoints: (SPE-RGS), (MPE-RGS), (MPE-CLS)

  [Descriptions]
  - SPE: single premise entailment approach
  - MPE: multiple premises entailment approach
  - CLS: classifcation model
  - RGS: regression model

1. Make sure you download the pretrained checkpoints for SS, RTE model.

2. Run demo.py, i.e., python demo.py Make sure your gpu is available.

3. Test the model with your own claim.

• For evaluation, run eval_pipeline.py, i.e., eval_pipeline.py

  python eval_pipeline.py --dr_pipeline <id> --ss_pipeline <id> --rte_pipeline <id>
  

• Various combinations can be implemented as followed:

  python eval_pipeline.py --dr_pipeline 2 --ss_pipeline 0 --rte_pipeline 2
  

1. DR

• The pipelines are loaded from dr/document_retrieval.py or eval_pipeline.py

• SimpleDR and SimpleDR2 reduce time for document retrieval.

  Id	Model	Description
  0	DocumentRetrieval	Loading wiki document titles and texts using wiki API
  1	SimpleDR	Using pre-retrieved wiki document texts
  2	SimpleDR2	Using pre-retrieved wiki document titles and texts

2. SS

• The pipelines are loaded from pipelines/ss_org.py and pipelines/ss_knn.py respectively.

  Id	Model	Description
  0	org	Unigram similarity approach
  1	KNN	K-nearest neighbors apporach

3. RTE

• The pipelines are loaded from pipelines/rte.py

  Id	Model	Accuracy(%)
  0	SPE-CLS	64.67
  1	MPE(noised)-CLS	76.79
  2	MPE(noised)-RGS	67.60
  3	SPE-RGS	54.93

• You can remove noise by --remove_noise option.