Code released by Huijuan Xu (Boston University).

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. Our model learns a fine-grained similarity metric for retrieval and uses visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task.

Our code is released under the MIT License (refer to the LICENSE file for details).

If you find our paper useful in your research, please consider citing:

@inproceedings{xu2019multilevel,
title={Multilevel Language and Vision Integration for Text-to-Clip Retrieval.},
author={Xu, Huijuan and He, Kun and Plummer, Bryan A. and Sigal, Leonid and Sclaroff,
Stan and Saenko, Kate},
booktitle={AAAI},
year={2019}
}

1. Installation
2. Preparation
3. Train Proposal Network
4. Extract Proposal Features
5. Training
6. Testing

1. Clone the Text-to-Clip_Retrieval repository.

   git clone --recursive [email protected]:VisionLearningGroup/Text-to-Clip_Retrieval.git

2. Build Caffe3d with pycaffe (see: Caffe installation instructions).

   Note: Caffe must be built with Python support!

cd ./caffe3d

# If have all of the requirements installed and your Makefile.config in
  place, then simply do:
make -j8 && make pycaffe

3. Build lib folder.

   cd ./lib    
   make

1. We convert the orginal data annotation files into json format.

   # train data json file
   caption_gt_train.json 
   # test data json file
   caption_gt_test.json

2. Download the videos in Charades dataset and extract frames at 25fps.

1. Generate the pickle data for training proposal network model.

   cd ./preprocess
   # generate training data
   python generate_roidb_modified_freq1.py

2. Download C3D classification pretrain model to ./pretrain/ .

3. In root folder, run proposal network training:

   bash ./experiments/train_rpn/script_train.sh

4. We provide one set of trained proposal network model weights.

1. In root folder, extract proposal features for training data and save as hdf5 data.

   bash ./experiments/extract_HDF_for_LSTM/script_test.sh

1. In root folder, run:

   bash ./experiments/Text_to_Clip/script_train.sh

1. Generate the pickle data for testing the Text_to_Clip model.

   cd ./preprocess
   # generate test data
   python generate_roidb_modified_freq1_full_retrieval_test.py

2. Download one sample model to ./experiments/Text_to_Clip/snapshot/ .

   One Text_to_Clip model on Charades-STA dataset is provided in: caffemodel .

   The provided model has Recall@1 (tIoU=0.7) score ~15.6% on the test set.

3. In root folder, generate the similarity scores on the test set and save as pickle file.

   bash ./experiments/Text_to_Clip/test_fast/script_test.sh 

4. Get the evaluation results.

   cd ./experiments/Text_to_Clip/test_fast/evaluation/
   bash bash.sh