Wele Gedara Chaminda Bandara, and Vishal M. Patel

For more information, please see our

• Paper: CVPR-2022-Open-Access or arxiv.
• Poster: view here
• Video Presentation: view here
• Presentation Slides: download here

Setup a virtual conda environment using the provided environment.yml file or requirements.txt.

conda env create --name HyperTransformer --file environment.yaml
conda activate HyperTransformer

or

conda create --name HyperTransformer --file requirements.txt
conda activate HyperTransformer

We use three publically available HSI datasets for experiments, namely

1. Pavia Center scene Download the .mat file here, and save it in "./datasets/pavia_centre/Pavia_centre.mat".
2. Botswana datasetDownload the .mat file here, and save it in "./datasets/botswana4/Botswana.mat".
3. Chikusei dataset Download the .mat file here, and save it in "./datasets/chikusei/chikusei.mat".

We use Wald's protocol to generate LR-HSI and PAN image. To generate those cubic patches,

1. Run process_pavia.m in ./datasets/pavia_centre/ to generate cubic patches.
2. Run process_botswana.m in ./datasets/botswana4/ to generate cubic patches.
3. Run process_chikusei.m in ./datasets/chikusei/ to generate cubic patches.

We use two stage procedure to train our HyperTransformer.

We first train the backbone of HyperTrasnformer and then fine-tune the MHFA modules. This way we get better results and faster convergence instead of training whole network at once.

Use the following codes to pre-train HyperTransformer on the three datasets.

1. Pre-training on Pavia Center Dataset:

   Change "train_dataset" to "pavia_dataset" in config_HSIT_PRE.json.

   Then use following commad to pre-train on Pavia Center dataset. python train.py --config configs/config_HSIT_PRE.json.

2. Pre-training on Botswana Dataset: Change "train_dataset" to "botswana4_dataset" in config_HSIT_PRE.json.

   Then use following commad to pre-train on Pavia Center dataset. python train.py --config configs/config_HSIT_PRE.json.

3. Pre-training on Chikusei Dataset:

   Change "train_dataset" to "chikusei_dataset" in config_HSIT_PRE.json.

   Then use following commad to pre-train on Pavia Center dataset. python train.py --config configs/config_HSIT_PRE.json.

Next, we fine-tune the MHFA modules in HyperTransformer starting from pre-trained backbone from the previous step.

1. Fine-tuning MHFA on Pavia Center Dataset:

   Change "train_dataset" to "pavia_dataset" in config_HSIT.json.

   Then use the following commad to train HyperTransformer on Pavia Center dataset.

   Please specify path to best model obtained from previous step using --resume. python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/pavia_dataset/N_modules\(4\)/best_model.pth.

2. Fine-tuning on Botswana Dataset:

   Change "train_dataset" to "botswana4_dataset" in config_HSIT.json.

   Then use following commad to pre-train on Pavia Center dataset.

   python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/botswana4/N_modules\(4\)/best_model.pth.

3. Fine-tuning on Chikusei Dataset:

   Change "train_dataset" to "chikusei_dataset" in config_HSIT.json.

   Then use following commad to pre-train on Pavia Center dataset.

   python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/chikusei_dataset/N_modules\(4\)/best_model.pth.

You can download trained models and final prediction outputs through the follwing links for each dataset.

1. Pavia Center: Download here
2. Botswana: Download here
3. Chikusei: Download here

If you find our work useful, please consider citing our paper.

@InProceedings{Bandara_2022_CVPR,
    author    = {Bandara, Wele Gedara Chaminda and Patel, Vishal M.},
    title     = {HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {1767-1777}
}