Anonymized github repository for NAACL 2022 submission
Install the conda virtual environment by:
conda env create -f env.ymlDownload nltk's punkt model (for response generation evaluation) by:
python -c "import nltk; nltk.download('punkt')"Download the dataset from repository via git-lfs. Run the script rearrange.sh to rearrange the data folder in the following format.
|-- images # scene images
| |-- cloth_store_1_1_1.png
| |-- cloth_store_1_1_2.png
| `-- ...
|-- jsons # bbox and scene jsons
| |-- cloth_store_1_1_1_bbox.json
| |-- cloth_store_1_1_1_scene.json
| `-- ...
|-- fashion_prefab_metadata_all.json # metadata (fashion)
|-- furniture_prefab_metadata_all.json # metadata (furniture)
|-- simmc2_dials_dstc10_dev.json # dialogue data (dev)
|-- simmc2_dials_dstc10_devtest.json # dialogue data (devtest)
|-- simmc2_dials_dstc10_train.json # dialogue data (train)
|-- simmc2_dials_dstc10_dev_retrieval_candidate.json # retrieval data (dev)
`-- simmc2_dials_dstc10_devtest_retrieval_candidate.json # retrieval data (devtest)
NOTE: Some of the scene images are corrupted and therefore ignored. We do not make use of images in this model other than getting image size.
./data/images/cloth_store_1416238_woman_4_8.png
./data/images/cloth_store_1416238_woman_19_0.png
./data/images/cloth_store_1416238_woman_20_6.png
Since our model is jointly trained on all tasks, we only need a single model for all subtasks. Download the model parameters by one of the following methods:
- Download from Google Drive: checkpoint-22000.zip
- Download with
gdown
gdown --id 1ffPkx1bcJrYL7nN88FCXDs5HrUc_SJhJFor our model input, preprocess the datasets to reformat the data.
Make sure to download simmc2-data into ./data before launching the code .
- Move into
scripts, run the following command.
python convert.py \
--input_path_json=<YOUR INPUT PATH JSON> \
--output_path_predict=<YOUR OUTPATH PREDICT> \
--output_path_target=<YOUR OUTPATH TARGET> \
--object_special_token_item2id=item2id.json \
--scene_json_folder=../data/jsons \
--image_folder ../data/images \For devtest dataset,
python convert.py \
--input_path_json=../data/simmc2_dials_dstc10_devtest.json \
--output_path_predict=../data_object_special/simmc2_dials_dstc10_devtest_predict.txt \
--output_path_target=../data_object_special/simmc2_dials_dstc10_devtest_target.txt \
--object_special_token_item2id=item2id.json \
--scene_json_folder=../data/jsons \
--image_folder=../data/images For teststd dataset without target(label) file,
python convert.py \
--input_path_json=../data/simmc2_dials_dstc10_teststd_public.json \
--output_path_predict=../teststd_data/teststd_predict.txt \
--object_special_token_item2id=item2id.json \
--scene_json_folder=../data/jsons \
--image_folder=../data/images \
--with_target=0Since our model is jointly trained on all subtasks, the additional target files are needed.
e.g simmc2_dials_dstc10_train_disambiguation_label.txt, simmc2_dials_dstc10_train_response.txt for disambiguation-task and retrieval-task, respectively. These are already uploaded in the directory data_object_special.
Our model is jointly trained with losses from each tasks based on BART.
Make sure to download simmc2-data into ./data before training: https://github.com/facebookresearch/simmc2/tree/main/data
- Move into
scripts, Run training.
bash run_bart_multi_task.shor
python run_bart_multi_task.py \
--add_special_tokens=../data_object_special/simmc_special_tokens.json \
--item2id=./item2id.json \
--train_input_file=../data_object_special/simmc2_dials_dstc10_train_predict.txt \
--train_target_file=../data_object_special/simmc2_dials_dstc10_train_target.txt \
--disambiguation_file=../data_object_special/simmc2_dials_dstc10_train_disambiguation_label.txt \
--response_file=../data_object_special/simmc2_dials_dstc10_train_response.txt \
--eval_input_file=../data_object_special/simmc2_dials_dstc10_devtest_predict.txt \
--eval_target_file=../data_object_special/simmc2_dials_dstc10_devtest_target.txt \
--output_dir=../multi_task/model \
--train_batch_size=12 \
--output_eval_file=../multi_task/model/report.txt \
--num_train_epochs=10 \
--eval_steps=3000 \
--warmup_steps=8000 All tasks can be evaluated with the same model parameters.
NOTE: For teststd split, input preprocessing instructions and preprocessed dataset can be found under teststd_data directory along with README.md.
bash run_bart_multi_task_disambiguation.shor
python run_bart_multi_task_disambiguation.py \
--path_output=../devtest_results/dstc10-simmc-devtest-pred-subtask-1.json \
--prompts_from_file=../data_object_special/simmc2_dials_dstc10_devtest_predict.txt \
--disambiguation_file=../data_object_special/simmc2_dials_dstc10_devtest_inference_disambiguation.json \
--item2id item2id.json \
--add_special_tokens=../data_object_special/simmc_special_tokens.json \
--model_dir=<YOUR MODEL CHECKPOINTS> Disambiguation file, simmc2_dials_dstc10_devtest_inference_disambiguation.json contains information on dialogue and turn index.
bash run_bart_multi_task_mm_dst.shor
python run_bart_multi_task_mm_dst.py \
--prompts_from_file=../data_object_special/simmc2_dials_dstc10_devtest_predict.txt \
--path_output=../devtest_results/dstc10-simmc-devtest-pred-subtask-3.txt \
--item2id=item2id.json \
--add_special_tokens=../data_object_special/simmc_special_tokens.json \
--model_dir=<YOUR MODEL CHECKPOINTS>This script creates a line-by-line *.txt prediction. To parse the line-by-line results dstc10-simmc-devtest-pred-subtask-3.txt into *subtask-4-generation.json format, use the following command in the directory preprocessing_data.
python convert_mm_dst_to_response.py \
--input_path_text=../devtest_results/dstc10-simmc-devtest-pred-subtask-3.txt \
--dialog_meta_data=../data_object_special/simmc2_dials_dstc10_devtest_inference_disambiguation.json \
--output_path_json=../devtest_results/dstc10-simmc-devtest-pred-subtask-4-generation.jsonbash run_bart_multi_task_retrieval.shor
python run_bart_multi_task_retrieval.py \
--path_output=../devtest_results/dstc10-simmc-devtest-pred-subtask-4-retrieval.json \
--prompts_from_file=../data_object_special/simmc2_dials_dstc10_devtest_predict.txt \
--candidate_file=../data_object_special/simmc2_dials_dstc10_devtest_retrieval.json \
--item2id item2id.json \
--add_special_tokens=../data_object_special/simmc_special_tokens.json \
--model_dir=<YOUR MODEL CHECKPOINTS>Candidate file, simmc2_dials_dstc10_devtest_retrieval.json contains the reformatted candidates, dialogue and turn index.
@article{kottur2021simmc,
title={SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations},
author={Kottur, Satwik and Moon, Seungwhan and Geramifard, Alborz and Damavandi, Babak},
journal={arXiv preprint arXiv:2104.08667},
year={2021}
}
@article{lewis2019bart,
title={Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension},
author={Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Ves and Zettlemoyer, Luke},
journal={arXiv preprint arXiv:1910.13461},
year={2019}
}
@article{radford2021learning,
title={Learning transferable visual models from natural language supervision},
author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and others},
journal={arXiv preprint arXiv:2103.00020},
year={2021}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent