factvc's Introduction

FactVC: Factual consistency for Video Captioning

This repository contains the data and code for the paper "Models See Hallucinations: Evaluating the Factuality in Video Captioning".

File Structure

FactVC-main/
├── data/
│   ├── activitynet/
│   │   ├── videos/         # sampled ActivityNet videos
│   │   ├── frames/         # extracted video frames
│   │   ├── captions/       # ground-truth and model-generated captions
│   │   ├── vids.txt        # video ids
│   │   └── factuality_annotation.json  # human factuality annotation
│   ├── youcook2/
│   │   ├── videos/         # sampled YouCook2 videos
│   │   ├── frames/         # extracted video frames
│   │   ├── captions/       # ground-truth and model-generated captions
│   │   ├── vids.txt        # video ids
│   │   └── factuality_annotation.json  # human factuality annotation
│   └── extract_frames.py
├── metric/
│   ├── clip/
│   ├── emscore/
│   └── factvc_corr.py      # code to compute FactVC score and correlation
└── pretrained_models
    └── factvc_video.pth    # our pretrained metric model

Usage

First, download the sampled ActivityNet videos and YouCook2 videos and unzip them into corresponding folders. Download the pretrained FactVC metric model and put it under pretrained_models/ folder.

Then, extract video frames at 1fps (used for computing FactVC metric scores):

cd data/
python extract_frames.py --dataset activitynet
python extract_frames.py --dataset youcook2

Now, you can compute the FactVC scores and the correlation between FactVC score and human annotation:

cd metric/
python factvc_corr.py --dataset activitynet
python factvc_corr.py --dataset youcook2

Acknowledgements

We acknowledge the EMScore project that we based on our work

factvc's People

Contributors

Watchers

factvc's Issues

Issue loading the model

Hi! I got this error when I am trying to load the model factvc_video.pth.
RuntimeError: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl
Could you tell me how to fix this?

Thanks for reading my message and I am looking forward to your reply

Recommend Projects