This repository is the Pytorch implementation of our paper "Maria: A Visual Experience Powered Conversational Agent" in ACL 2021.

In this paper, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e., text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator.

Coming soon!

• Maria: A Visual Experience Powered Conversational Agent
  • Summary
  • Dependencies
  • Usage
    • Text-to-Image Retrieval Model
    • Bottom-up Detector Model
    • Dialog Generation Model
  • Citation
  • Acknowledgment

• python 3.7

• pytorch 1.4.0

• Ubuntu 18.04

Please download the Reddit data from google drive here.

We will use the Open Images images as candidate images for retrievel. Refer to here to download the images first. You can build the image index with the appropriate size (500,000 in our experiments) as needed.

If you already have Open Images dataset on disk. Save them as

data
|-- open_images
    |-- images
         |-- 14928b4f367c217e.jpg
         |-- 289d643a8761aa83.jpg
         |-- ......

Please refer to retrieval_model/README.md

Please refer to detector_model/README.md

Please refer to dialog_model/README.md

If you find this paper helps your research, please kindly consider citing our paper in your publications.

@inproceedings{liang2021maria,
   title={Maria: A Visual Experience Powered Conversational Agent},
   author={Liang, Zujie and 
           Hu, Huang and 
           Xu, Can and 
           Tao, Chongyang and 
           Geng, Xiubo and 
           Chen, Yining and 
           Liang, Fan and 
           Jiang, Daxin},
   booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)},
   year={2021}
}

Special thanks to the authors of OSCAR, vokenization, and py-bottom-up-attention.