DramaQA dataset is built upon the TV drama Another Miss Oh and it contains 16,191 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered visual annotations and coreference resolved scripts.

• Dataset Overview

Dual Matching Multistream model effectively learns character-centered representations of video to answer questions about the video.

• Model Overview

We use python3 (3.5.2), and python2 is not supported. We use PyTorch (1.1.0), though tensorflow-gpu is necessary to launch tensorboard.

python packages: fire for commandline api

git clone --recurse-submodules (this repo)
cd $REPO_NAME/code
(use python >= 3.5)
pip install -r requirements.txt
python -m nltk.downloader 'punkt'

Place the data folder at $REPO_NAME/data.

Download DramaQA dataset from here and place the data folder like following structure.

code/
  cli.py
  train.py
  evaluate.py
  infer.py
  ...
data/
  AnotherMissOh/
    AnotherMissOh_images/
      $IMAGE_FOLDERS
    AnotherMissOh_QA/
      AnotherMissOhQA_train_set.json
      AnotherMissOhQA_val_set.json
      AnotherMissOhQA_test_set.json
    AnotherMissOh_Visual.json
    AnotherMissOh_script.json

cd code
python cli.py train

The above command will preprocess textual and visual input before it starts training process. Access the prompted tensorboard port to view basic statistics. At the end of every epoch, a checkpoint file will be saved on data/ckpt/OPTION_NAMES

For further configurations, take a look at code/config.py and fire.

cd code
python cli.py evaluate --ckpt_name=$CKPT_NAME

Substitute CKPT_NAME to your prefered checkpoint file. e.g. --ckpt_name=model_name_dmm_ckpt_3/loss_0.4818_epoch_15

Challenge participants should submit their outcome with this command.

python cli.py infer --ckpt_name=$CKPT_NAME

The above command will save the outcome at the prompted location.

See the Troubleshooting page and submit a new issue or contact us if you cannot find an answer.

To contact us, send an email to [email protected]