VQA using the Differentiable Forth Interpreter (∂4)

We apply ∂4 to a visual question answering problem, and they jointly learned using the CLEVR dataset end-to-end. Here for more details.

Installation

Dependencies

∂4 interpreter

Python 3

Pytorch 1.10.0

with

• Tensorflow 0.11.0

pip3 install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0-cp35-cp35m-linux_x86_64.whl

Use

Dataset

Generate CLEVR dataset form here.

Use this template for generating questions.

And save the rendered images and the generated CLEVR_questions.json to vqa/data directory.

Note

We edit ∂4: extensible_dsm.py, line 275. We changed the type into float32: create_alg_op_matrixret = np.zeros([size, size,size], dtype=np.float32).

Running the experiment

Extract features

python3 scripts/extract_features.py \
--input_image_dir data/images\
--output_h5_file data/train_features.h5

Process questions

Use this vocal.json for vocabs.

python3 scripts/preprocess_questions.py  \
--input_questions_json data/CLEVR_questions.json \
--input_vocab_json data/vocab.json \
--output_h5_file data/train_questions.h5

Training

Use this vocal.json for vocabs.

python3  scripts/trainer.py