Purpose: We aim to provide a summary of diffusion model-based image processing, including restoration, enhancement, coding, and quality assessment. More papers will be summarized.
Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, Zhibo Chen
University of Science and Technology of China (USTC), National University of Singapore (NUS), Microsoft Research Asia (MSRA), Eastern Institute of Technology (EIT)
Brief intro: The survey for diffusion model-based IR has been released.
- 2023-09-19: Updated new related works before 15/09/2023 in this GitHub.
- 2023-11-24: Updated new related works before 10/11/2023 in this GitHub.
- 2023-12-25: Updated new related works before 25/12/2023 in this GitHub.
- 2024-01-25: Updated new related works before 25/01/2024 in this GitHub.
- 2024-03-25: Updated new related works before 25/03/2024 in this GitHub.
- 2024-04-25: Updated new related works before 25/04/2024 in this GitHub.
- 2024-06-25: Updated new related works before 25/06/2024 in this GitHub.
- 2024-08-25: Updated new related works before 25/08/2024 in this GitHub.
📌 About new works. If you want to incorporate your studies (e.g., the link of paper or project) on diffusion model-based image processing in this repository. Welcome to raise an issue or email us. We will incorporate it into this repository and our survey report as soon as possible.
- Survey for diffusion model-based Image Restoration (Arxiv version is released)
- Summary for diffusion model-based Image/Video Compression
- Summary for diffusion model-based Quality Assessment
- Diffusion model for Image Super resolution
- Diffusion model for Image Restoration
- Diffusion model for Image Inpainting
- Diffusion model for Image Shadow Removal
- Diffusion model for Image Denoising
- Diffusion model for Image Dehazing
- Diffusion model for Image Deblurring
- Diffusion model for Medical IR
- Diffusion model for Low-Light Enchancement
- Diffusion model for other tasks
- Benchmark Datasets
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
RePaint | RePaint: Inpainting using denoising diffusion probabilistic models | Andreas Lugmayr | Zero-shot | CVPR2022 | Image Inpainting | |
CoPaint | Towards coherent image inpainting using denoising diffusion implicit models | Guanhua Zhang | Zero-shot | PrePrint'23 | Image Inpainting | |
PVA | Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention | Jianjin Xu | Zero-shot | PrePrint'23 | Image Inpainting |
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
BCDiff | Boundary-Aware Divide and Conquer: A Diffusion-based Solution for Unsupervised Shadow Removal | Lanqing Guo | Unsupervised | ICCV 2023 | Image Shadow Removal | |
ShadowDiffusion | Shadowdiffusion: When degradation prior meets diffusion model for shadow removal | Lanqing Guo | Supervised | CVPR2023 | Image Shadow Removal | |
DeS3 | DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence | Yeying Jin | Supervised | AAAI'24 | Image Shadow Removal | |
LFG-Diffusion | Latent Feature-Guided Diffusion Models for Shadow Removal | Kangfu Mei | Supervised | WACV 2024 | Image Shadow Removal | |
Deshadow-Anything | Deshadow-Anything: When Segment Anything Model Meets Zero-shot shadow removal | Xiao Feng Zhang | Supervised | Preprint'23 | Image Shadow Removal |
Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|
Diffusion model for generative image denoising | Yutong Xie | Supervised | Preprint'23 | Image Denoising | |
Score-Based Diffusion Models as Principled Priors for Inverse Imaging | Berthy T. Feng | Zero-shot | ICCV 2023 | Image Denoising, Image Deblurring | |
Real-World Denoising via Diffusion Model | Cheng Yang | Supervised | PrePrint'23 | Image Denoising |
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
HazeDDPM | High-quality Image Dehazing with Diffusion Model | Hu Yu | Supervised | Preprint'23 | Image Dehazing | |
-- | Frequency Compensated Diffusion Model for Real-scene Dehazing | Jing Wang | Supervised | Preprint'23 | Image Dehazing |
Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|
Deblurring via stochastic refinement | Jay Whang | Supervised | CVPR2022 | Image Deblurring | |
Score-Based Diffusion Models as Principled Priors for Inverse Imaging | Berthy T. Feng | Zero-shot | ICCV 2023 | Image Denoising, Image Deblurring | |
BlindDPS:Parallel diffusion models of operator and image for blind inverse problems | Hyungjin Chung | Zero-shot | CVPR2023 | Blind Deblurring | |
HI-Diff:Hierarchical Integration Diffusion Model for Realistic Image Deblurring | Zheng Chen | Supervised | Prepreint'23 | Image Deblurring | |
Swintormer:Efficient Image Deblurring Networks based on Diffusion Models | Kang Chen | Supervised | Prepreint'24 | Image Deblurring |
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
MCG | Improving diffusion models for inverse problems using manifold constraints | Hyungjin Chung | Zero-shot | NeurIPS 2022 | CT Reconstruction | |
-- | Solving inverse problems in medical imaging with score-based generative models | Yang Song | Zero-shot | ICLR 2022 | CT Reconstruction | |
AdaDiff | Adaptive diffusion priors for accelerated mri reconstruction | Alper Güngör | Supervised | Preprint'23 | MRI Reconstruction | |
HFS-SDE | High-Frequency Space Diffusion Models for Accelerated MRI | Chentao Cao | Supervised | Preprint'22 | MRI Reconstruction | |
SWORD | Stage-by-stage Wavelet Optimization Refinement Diffusion Model for Sparse-View CT Reconstruction | Kai Xu | Supervised | Preprint'23 | MRI Reconstruction | |
FDB | Learning Fourier-Constrained Diffusion Bridges for MRI Reconstruction | Muhammad U. Mirza | Supervised | Preprint'23 | MRI Reconstruction |
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
PyDiff | Pyramid Diffusion Models For Low-light Image Enhancement | Dewei Zhou | Supervised | Preprint'23 | Low-light Enhancemnent | |
DiffLL | Lowlight image enhancement with wavelet-based diffusion models | Hai Jiang | Supervised | Preprint'23 | Low-light Enhancemnent | |
ExploreDiffusion | ExposureDiffusion: Learning to Expose for Low-light Image Enhancement | Yufei Wang | Supervised | ICCV 2023 | Low-light Enhancemnent | |
Diff-Retinex | Rethinking Low-light Image Enhancement with A Generative Diffusion Model | Xunpeng Yi | Supervised | ICCV 2023 | Low-light Enhancemnent | |
CLE Diffusion | CLE Diffusion: Controllable Light Enhancement Diffusion Model | Yuyang Yin | Supervised | Preprint'23 | Low-light Enhancemnent | |
DiffLLE | DiffLLE: Diffusion-guided Domain Calibration for Unsupervised Low-light Image Enhancement | Shuzhou Yang | Supervised | Preprint'23 | Low-light Enhancemnent | |
LLDiffusion | LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement | Tao Wang | Supervised | Preprint'23 | Low-light Enhancemnent | |
Entropy-SDE | Equipping Diffusion Models with Differentiable Spatial Entropy for Low-Light Image Enhancement | Wenyi Lian | Supervised | CVPRW 2024 | Low-light Enhancemnent |
Dataset | Task | Usage | Year |
---|---|---|---|
DIV2K | Image Super-resolution | Training,Testing | 2017 |
Flickr2K | Image Super-resolution | Training | 2017 |
Set5 | Image Super-resolution | Testing | 2012 |
Set14 | Image Super-resolution | Testing | 2012 |
BSD100 | Image Super-resolution | Testing | 2012 |
Manga109 | Image Super-resolution | Testing | 2015 |
Urban100 | Image Super-resolution | Testing | 2015 |
OST300 | Image Super-resolution | Testing | 2018 |
DIV8K | Image Super-resolution | Training,Testing | 2019 |
RealSR | Image Super-resolution | Training,Testing | 2019 |
DRealSR | Image Super-resolution | Training,Testing | 2020 |
GoPro | Image Deblurring | Training,Testing | 2017 |
HIDE | Image Deblurring | Training,Testing | 2019 |
RealBlur | Image Deblurring | Training,Testing | 2020 |
Kodak | Image Denoising | Testing | 1999 |
CBSD68 | Image Denoising | Testing | 2001 |
McMaster | Image Denoising | Testing | 2011 |
ImageNet | Image Classification | Training,Testing | 2010 |
ImageNet1k | Image Classification | Testing | 2020 |
LSUN | Image Classification | Training,Testing | 2015 |
Places365 | Image Classification | Training,Testing | 2019 |
LFW | Face Generation | Training | 2008 |
FFHQ | Face Generation | Training | 2019 |
Celeba-HQ | Face Generation | Training | 2018 |
AFHQ | Face Generation | Training | 2020 |
CelebA | Face Generation | Training | 2015 |
ISTD | Image Shadow Removal | Training,Testing | 2018 |
SRD | Image Shadow Removal | Training,Testing | 2017 |
CSD | Image Desnowing | Training,Testing | 2021 |
Snow100k | Image Desnowing | Training,Testing | 2017 |
SRRS | Image Desnowing | Training,Testing | 2020 |
RainDrop | Image Deraining | Training | 2018 |
RainDropClarity | Image Deraining | Training,Testing | 2024 |
Outdoor-Rain | Image Deraining | Training,Testing | 2019 |
DDN-data | Image Deraining | Training,Testing | 2017 |
SPA-data | Image Deraining | Training,Testing | 2019 |
Rain100H | Image Deraining | Training,Testing | 2017 |
Rain100L | Image Deraining | Training,Testing | 2017 |
Haze-4K | Image Dehazing | Training | 2021 |
Dense-Haze | Image Dehazing | Training | 2019 |
RESIDE | Image Dehazing | Training | 2019 |
Model | Paper | First Author | Venue | Topic | Project |
---|---|---|---|---|---|
DifFIQA | DifFIQA: Face Image Quality Assessment Using Denoising Diffusion Probabilistic Models | Žiga Babnik | Preprint'23 | Image quality assessment | |
PFD-IQA | Feature Denoising Diffusion Model for Blind Image Quality Assessment | Xudong Li | Preprint'24 | Image quality assessment |
If this work is helpful to you, we expect you can cite this work and star this repo. Thanks.
@article{li2023diffusion,
title={Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey},
author={Li, Xin and Ren, Yulin and Jin, Xin and Lan, Cuiling and Wang, Xingrui and Zeng, Wenjun and Wang, Xinchao and Chen, Zhibo},
journal={arXiv preprint arXiv:2308.09388},
year={2023}
}