Official code implementation for the paper S² Transformer for Imgae Captioning
Pengpeng Zeng, Haonan Zhang, Jingkuan Song, and Lianli Gao

• Environment setup
• Data Preparation
• Training
• Evaluation
• Reference and Citation
• Acknowledgements

Clone this repository and create the m2release conda environment using the environment.yml file:

conda env create -f environment.yaml
conda activate m2release

Then download spacy data by executing the following command:

python -m spacy download en_core_web_md

• Annotation. Download the annotation file m2_annotations [1]. Extract and put it in the project root directory.
• Feature. Download processed image features ResNeXt-101 and ResNeXt-152 features [2] (code 9vtB), put it in the project root directory.

Update: Image features on OneDrive

Run python train_transformer.py using the following arguments:

For example, to train the model, run the following command:

python train_transformer.py --exp_name S2 --batch_size 50 --m 40 --head 8 --features_path /path/to/features --num_clusters 5

or just run:

bash train.sh

Run python test_transformer.py to evaluate the model using the following arguments:

python test_transformer.py --batch_size 10 --features_path /path/to/features --model_path /path/to/saved_transformer_models/ckpt --num_clusters 5

We provide checkpoint here, you will get the following results (second row):

We also report the performance of our model on the online COCO test server with an ensemble of four S² models. The detailed online test code can be obtained in this repo.

[1] Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020). Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[2] Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15465–15474, 2021.

@inproceedings{S2,
  author    = {Pengpeng Zeng* and
               Haonan Zhang* and
               Jingkuan Song and 
               Lianli Gao},
  title     = {S2 Transformer for Image Captioning},
  booktitle = {IJCAI},
  pages     = {1608--1614}
  year      = {2022}
}

Thanks Zhang et.al for releasing the visual features (ResNeXt-101 and ResNeXt-152). Our code implementation is also based on their repo.
Thanks for the original annotations prepared by M² Transformer, and effective visual representation from grid-feats-vqa.