rayabhisek123 / cfat Goto Github PK

Abhisek Ray¹, Gaurav Kumar¹, Maheshkumar H. Kolekar¹

¹Indian Institute of Technology Patna, India

• ✅ 2024-03-24: Release the first version of the paper at Arxiv.
• ✅ 2024-03-24: Release the supplementary material of the paper at Arxiv.
• ✅ 2024-03-24: Release the codes, models and results of CFAT.
• ✅ 2024-06-11: Update repo with CVPR version.
• (To do) Release the small (CFAT-S) and large (CFAT-L) versions of our model.
• (To do) Add the pre-trained model of CFAT for SRx4.
• (To do) Add a Replicate demo for CFAT model implementation.
• (To do) Release extensive code of CFAT for Multiple Image Restoration tasks.

Abstract: Transformer-based models have revolutionized the field of image super-resolution (SR) by harnessing their inherent ability to capture complex contextual features. The overlapping rectangular shifted window technique used in transformer architecture nowadays is a common practice in super-resolution models to improve the quality and robustness of image upscaling. However, it suffers from distortion at the boundaries and has limited unique shifting modes. To overcome these weaknesses, we propose a non-overlapping triangular window technique that synchronously works with the rectangular one to mitigate boundary-level distortion and allows the model to access more unique sifting modes. In this paper, we propose a Composite Fusion Attention Transformer (CFAT) that incorporates triangular-rectangular window-based local attention with a channel-based global attention technique in image super-resolution. As a result, CFAT enables attention mechanisms to be activated on more image pixels and captures long-range, multi-scale features to improve SR performance. The extensive experimental results and ablation study demonstrate the effectiveness of CFAT in the SR domain. Our proposed model shows a significant 0.7 dB performance improvement over other state-of-the-art SR architectures.

The triangular window mechanism that we proposed is beneficial not only in super-resolution tasks but also in various other computer vision applications that implement the rectangular window technique in their mainframe.

• Quantitative Results

Fig. Comparing performance (PSNR in dB) of various state-of-the-art models with CFAT.

• Qualitative Results

Fig. Visual Comparison of CFAT with other state-of-the-art methods.

• LAM Results

Fig. LAM results and corresponding Diffusion Index for CFAT and various SOTA methods.

• Requirements

  • Platforms: Ubuntu 20.04.2, cuda-11.2.0
  • Python 3.8.18, PyTorch == 2.1.0
  • Requirements: see requirements.txt

• Installation

# download code
git clone https://github.com/rayabhisek123/CFAT
cd CFAT
pip install -r requirements.txt
pip install basicsr
python setup.py develop

• Datasets

  Download the training datasets:

  • DIV2K
  • Flickr2K

  Download the testing datasets:

  • Set5, Set14, BSD100，Urban100
  • Manga109

Command to train CFAT after placing datasets at their respective repositories

CUDA_VISIBLE_DEVICES=0,1,2 python3 train.py

The inference results on benchmark datasets will be available soon.

Run the following command after placing the Pre-trained Models in the given repository

CUDA_VISIBLE_DEVICES=0,1 python3 test.py

**Note: For different configurations, change the argument values. We will update the corresponding configuration files(.yml) soon.

Fig. Quantitative comparison of the CFAT with various state-of-the-art SR methods. Red: Best & Green: Second Best.

@article{ray2024cfat,
  title={CFAT: Unleashing TriangularWindows for Image Super-resolution},
  author={Ray, Abhisek and Kumar, Gaurav and Kolekar, Maheshkumar H},
  journal={arXiv preprint arXiv:2403.16143},
  year={2024}
}

Some parts of this code are adapted from:

• HAT
• SwinIR

We thank the authors for sharing codes of their great works.

This project is licensed under the MIT License and was originally developed by @abhisek-ray.

If you have any questions, please email [email protected] to discuss with the authors.