We inspect the effectiveness of PEFT methods on the Q-Former and LLM layer for Visual Reasoning Tasks.

Visual language models have recently demonstrated enhanced capabilities in visual reasoning tasks by employing external modules upon language models for visual language alignment. InstructBLIP uses a Q-Former and a projection layer to convert input image embeddings into soft visual prompts to enhance the instruction-following capabilities of large language models (LLMs). Although fine-tuning InstructBLIP has shown great results in downstream tasks, previous works have been restrictive, only full fine-tuning the Q-Former, while freezing the LLM.

In this work, we investigate the performance of the PEFT method, LoRA, on both the Q-Former and the base LLMs, specifically Flan-T5-XL and Vicuna-7B, using visual reasoning benchmarks ScienceQA and IconQA. We observe that, when the LLM is frozen, training the Q-Former with LoRA achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Furthermore, fine-tuning the LLM consistently result in better performances than InstructBLIP. Lastly, applying LoRA to both the LLM and the Q-Former surpasses the performance of only full fine-tuning the Q-Former while using less than 12% of the trainable parameters. These results highlight the effectiveness of applying PEFT to visual language models for visual reasoning tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.

• Install
• Train
• Citation
• Acknowledgement

1. Clone this repository and navigate to InstructBLIP_PEFT folder.

git clone https://github.com/AttentionX/InstructBLIP_PEFT.git
cd InstructBLIP_PEFT

2. Install Package

pip install -r requirements.txt

1. download ScienceQA dataset from https://scienceqa.github.io/
2. run scienceqa_data_preprocess.py

This will save preprocessed scienceQA dataset in /input/scienceqa/.

This is the Instruction Format for ScienceQA dataset.

<Image> Context: { {hint} {lecture} } Question: { {question} } Options: { {choices} } Answer: (a) { {answer} }

1. download multi-text-choice dataset from https://iconqa.github.io/
2. run iconqa_data_preprocess.py

This will save preprocessed scienceQA dataset in /input/iconqa/.

This is the Instruction Format for IconQA dataset.

<Image> Question: { {question} } Options: { {choices} }. Short answer: (a) { {answer} }

We train our model using a single A100 GPU.

Datasets must be placed in the location specified in the file lavis/config/datasets/{dataset_name}/defaults.yaml .

This is an example of dataset default.yaml file.

# lavis/config/datasets/scienceqa/default.yaml
datasets:
  scienceqa:
    # data_dir: ${env.data_dir}/datasets
    data_type: images # [images|videos|features]

    build_info:
      # Be careful not to append minus sign (-) before split to avoid itemizing
      annotations:
        train:
          storage: /input/scienceqa/scienceqa_train.json
        val:
          storage: /input/scienceqa/scienceqa_val.json
        test:
          storage: /input/scienceqa/scienceqa_test.json
      images:
        storage: /input
        train:
          storage: /input
        val:
          storage: /input
        test:
          storage: /input

In this case, dataset json files (scienceqa_train.json, scienceqa_test.json and scienceqa_val.json) should be located at /input/scienceqa.
Images files should be located at input/scienceqa/images/train, input/scienceqa/images/test and input/scienceqa/images/val because of the content in json files.

This is the table for the ID for each experiements.

You can run experiment with this command.

bash run_scripts/instructblip/train/run_finetune_instructblip_experiments.sh {dataset_name} {experiment_id}

The result will be saved in /input/results/{dataset_name}/{experiment_id}. You can change this in sh file run_finetune_instructblip_experiments.sh.

For example, If you want to try experiment 15 for scienceqa, you can use this command.

bash run_scripts/instructblip/train/run_finetune_instructblip_experiments.sh scienceqa 15

• @Lightning-AI for lit-llama
• @FacebookResearch for the original LLaMA implementation
• @Salesforce for LAVIS

BSD 3-Clause License (from LAVIS)

Apache 2.0 License (From lit-llama)