A code for Unlearning Traces the Influential Training Data of Language Models @ ACL 2024 (main, long paper)
Authors:
Masaru Isonuma1,2 and Ivan Titov1,3
1University of Edinburgh
2University of Tokyo
3University of Amsterdam
Python==3.8
- Run the following command to install the required packages:
pip install -r requirements.txt
- Or you can reproduce the environment if you use conda:
conda env create -n untrac -f untrac.yml
conda activate untrac
- Train T5-XL-lm-adapt on the synthetic dataset A:
bash train_synthetic.sh
- Run Untrac; Unlearn each training dataset from the trained model and evaluate the unlearned model on the test dataset:
bash untrac_synthetic.sh
- Run Untrac-Inv; Unlearn the test dataset from the trained model and evaluate the unlearned model on each training dataset:
bash untrac-inv_synthetic.sh
- Run the following command to reproduce leave-dataset-out:
bash loo_synthetic.sh
-
Refer to
evaluate_synthetic.ipynb
to assess UnTrac and UnTrac-Inv. -
Replace
synthetic_train00_dataset
&synthetic_eval00_dataset
withsynthetic_train01_dataset
&synthetic_eval01_dataset
in the shell scripts if you want to switch to the synthetic dataset B.
- Preprocess the training datasets and test datasets; then pre-train OPT from scratch:
bash pretrain_opt.sh
- The downloaded datasets are stored in the
data
directory. Replacemodel_name_or_path
,train_dir
, andeval_dir
inuntrac_synthetic.sh
with the trained OPT model and the downloaded datasets.