Some important packages are listed as follows.

python == 3.7
pytorch == 1.13.1
transformers == 4.22.2
numpy == 1.21.6

We have provided the data of Twitter-15, Twitter-17, MVSA-Single and MVSA-Multiple. You don't need the images to run the code in the repo.

You can leverage the CoT prompting approach, as outlined in our paper, to utilize miniGPT-4 for knowledge generation, including image descriptions and rationales.

The training/eval scripts are very straightforward, and follow the same structure. Looking at main.py as a concrete example, all you need to do are to change the config.ini and run the code:

python main.py

If you found this paper useful, citing the paper and dataset would be greatly appreciated after the paper is accepted.