News | Introduction | Preparation | Demo | Acknowledgement | Statement

• [Feb 21 2024]: We have updated our evaluation code. Any advice are welcom!
• [Feb 7 2024]: Model weights are now available on both Google Drive and Baidu Disk.
• [Feb 6 2024]: Our paper now is available at arxiv.
• [Feb 2 2024]: We are excited to announce the release of our code and model checkpoint! Our dataset and training recipe will be update soon!

We are excited to introduce LHRS-Bot, a multimodal large language model (MLLM) that leverages globally available volunteer geographic information (VGI) and remote sensing images (RS). LHRS-Bot demonstrates a deep understanding of RS imagery and possesses the capability for sophisticated reasoning within the RS domain. In this repository, we will release our code, training framework, model weights, and dataset!

1. Clone this repository.

   git clone [email protected]:NJU-LHRS/LHRS-Bot.git
   cd LHRS-Bot

2. Create a new virtual enviroment

   conda create -n lhrs python=3.10
   conda activate lhrs

3. Install dependences and our package

   pip install -e .

• LLaMA2-7B-Chat

  • Automaticaly download:

    Our framework is designed to automatically download the checkpoint when you initiate training or run a demo. However, there are a few preparatory steps you need to complete:

    1. Request the LLaMA2-7B models from Meta website.

    2. After your request been processed, login to huggingface using your personal access tokens:

       huggingface-cli login
       (Then paste your access token and press Enter)

    3. Done!

  • Mannually download:

    • Download all the files from HuggingFace.

    • Change the following line of each file to your downloaded directory:

      • /Config/multi_modal_stage{1, 2, 3}.yaml

        ...
        text:
        	...
          path: ""  # TODO: Direct to your directory
        ...

      • /Config/multi_modal_eval.yaml

        ...
        text:
        	...
          path: ""  # TODO: Direct to your directory
        ...

• LHRS-Bot Checkpoints:

  Staeg1	Stage2	Stage3
  Baidu Disk, Google Drive	Baidu Disk, Google Drive	Baidu Disk, Google Drive

  • ⚠️ Ensure that the TextLoRA folder is located in the same directory as FINAL.pt. The name TextLoRA should remain unchanged. Our framework will automatically detect the version perceiver checkpoint and, if possible, load and merge the LoRA module.

  • Development Checkpoint:

    We will continually update our model with advanced techniques. If you're interested, feel free to download it and have fun :)

    Development
    Baidu Disk, Google Drive

• Online Web UI demo with gradio:

  python lhrs_webui.py \
       -c Config/multi_modal_eval.yaml \           # config file
       --checkpoint-path ${PathToCheckpoint}.pt \  # path to checkpoint end with .pt
       --server-port 8000 \                        # change if you need
       --server-name 127.0.0.1 \                   # change if you need
       --share                                     # if you want to share with other

• Command line demo:

  python cli_qa.py \
       -c Config/multi_modal_eval.yaml \                 # config file
       --model-path ${PathToCheckpoint}.pt \             # path to checkpoint end with .pt
       --image-file ${TheImagePathYouWantToChat} \       # path to image file (Only Single Image File is supported)
       --accelerator "gpu" \                             # change if you need ["mps", "cpu", "gpu"]
       --temperature 0.4 \
       --max-new-tokens 512

• Inference:

  • Classification

    python main_cls.py \
         -c Config/multi_modal_eval.yaml \                 # config file
         --model-path ${PathToCheckpoint}.pt \             # path to checkpoint end with .pt
         --data-path ${ImageFolder} \                      # path to classification image folder
         --accelerator "gpu" \                             # change if you need ["mps", "cpu", "gpu"]
         --workers 4 \
         --enabl-amp True \
         --output ${YourOutputDir}                         # Path to output (result, metric etc.)
         --batch-size 8 \

  • Visual Grounding

    python main_vg.py \
         -c Config/multi_modal_eval.yaml \                 # config file
         --model-path ${PathToCheckpoint}.pt \             # path to checkpoint end with .pt
         --data-path ${ImageFolder} \                      # path to image folder
         --accelerator "gpu" \                             # change if you need ["mps", "cpu", "gpu"]
         --workers 2 \
         --enabl-amp True \
         --output ${YourOutputDir}                         # Path to output (result, metric etc.)
         --batch-size 1 \                                  # It's better to use batchsize 1, since we find batch inference
         --data-target ${ParsedLabelJsonPath}              # is not stable.

  • Visual Question Answering

    python main_vqa.py \
         -c Config/multi_modal_eval.yaml \                 # config file
         --model-path ${PathToCheckpoint}.pt \             # path to checkpoint end with .pt
         --data-path ${Image} \                            # path to image folder
         --accelerator "gpu" \                             # change if you need ["mps", "cpu", "gpu"]
         --workers 2 \
         --enabl-amp True \
         --output ${YourOutputDir}                         # Path to output (result, metric etc.)
         --batch-size 1 \                                  # It's better to use batchsize 1, since we find batch inference
         --data-target ${ParsedLabelJsonPath}              # is not stable.
         --data-type "HR"                                  # choose from ["HR", "LR"]

• We gratitude to the following repositories for their wonderful works:
  • LLaVA
  • MiniGPT-4
  • LLaMA

• If you find our work is useful, please give us 🌟 in GitHub and consider cite our paper:

  @misc{2402.02544,
  Author = {Dilxat Muhtar and Zhenshi Li and Feng Gu and Xueliang Zhang and Pengfeng Xiao},
  Title = {LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model},
  Year = {2024},
  Eprint = {arXiv:2402.02544},
  }

• Licence: Apache