This repo is a collection of AWESOME papers/codes/blogs about Uncertainty and Deep learning, including papers, code, etc. Feel free to star and fork.

if you think we missed a paper, please send us an email at: gianni.franchi at ensta-paris.fr with the following subject awesome-uncertainty-deeplearning. (tell us where it is published, and send us a GitHub link and arxiv link if they are available)

• Contents
• Papers
  • Survey
  • Theory
  • Ensemble/Bayesian-Methods
  • Sampling/Dropout-based-Methods
  • Learning-loss-distributions/Auxiliary-network-Methods
  • Data-augmentation/Generation-based-Methods
  • Calibration
  • Prior-networks/Evidential-deep-learning
  • Deterministic-Uncertainty-Methods
  • Quantile-Regression/Predicted-Intervals
  • Applications
    • Classification and Semantic-Segmentation
    • Regression
    • Annomaly-detection and out of Distribution Dectection
• Datasets and Benchmarks
• Library
• Lectures and Tutorials
• Other Resources

Arxiv

• Ensemble deep learning: A review. [arxiv2021]
• A survey of uncertainty in deep neural networks.[arxiv2021]
• A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation [arxiv2021]

Conference

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications[AISafety2020 Workshop]

Journal

• Predictive inference with the jackknife+." [The Annals of Statistic(2021)]
• A review of uncertainty quantification in deep learning: Techniques, applications and challenges [Information Fusion 2021]
• A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective [ACM2021]
• Uncertainty in big data analytics: survey, opportunities, and challenges [Journal of Big Data2019]

Arxiv

• Bayesian Model Selection, the Marginal Likelihood, and Generalization [arxiv2022]
• Testing for Outliers with Conformal p-values [arxiv2021] [python]
• Efficient Gaussian Neural Processes for Regression [arxiv2021]
• DEUP: Direct Epistemic Uncertainty Prediction [arxiv2020]
• A higher-order swiss army infinitesimal jackknife [arxiv2019]
• With malice towards none: Assessing uncertainty via equalized coverage [arxiv2019]

Conference

• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• Neural Variational Gradient Descent [AABI2022]
• Bayesian Optimization with High-Dimensional Outputs [NIPS2021]
• Residual Pathway Priors for Soft Equivariance Constraints [NIPS2021]
• Dangers of Bayesian Model Averaging under Covariate Shift [NIPS2021] [Tensorflow]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR2021Workshop]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] [Pytorch]
• Evidential Deep Learning to Quantify Classification Uncertainty [NIPS2018] [Pytorch]
• On the accuracy of influence functions for measuring group effects [NIPS2018]
• To Trust Or Not To Trust A Classifier [NIPS2018] [python]

Journal

• Multivariate Uncertainty in Deep Learning [TNNLS2021]
• A General Framework for Uncertainty Estimation in Deep Learning [RAL2020]
• Adaptive nonparametric confidence sets [The Annals of Statistic(2006)]

Arxiv

• Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks [arxiv2022]
• Confident Neural Network Regression with Bootstrapped Deep Ensembles [arxiv2022] [Tensorflow]
• Repulsive Deep Ensembles are Bayesian [arxiv2021]
• Bayesian Neural Networks with Soft Evidence [arxiv2020] [Pytorch]
• On Batch Normalisation for Approximate Bayesian Inference [arxiv2020]
• Bayesian neural network via stochastic gradient descent [arxiv2020]
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [arxiv2020] [Pytorch]
• Deep Ensembles: A Loss Landscape Perspective [arxiv2019]
• Diversity with Cooperation: Ensemble Methods for Few-Shot Classification [arxiv2019]
• Ensemble Distribution Distillation [arxiv2018]

Conference

• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity [ICLR2022] [Pytorch]
• Masksembles for Uncertainty Estimation [CVPR2021] [Pytorch/Tensorflow]
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks [UAI2021]
• Learnable uncertainty under Laplace approximations [UAI2021]
• Real-time uncertainty estimation in computer vision via uncertainty-aware distribution distillation [WACV2021]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] [Pytorch]
• Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles [AAAI2020]
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification [NIPS2020]
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks [ICML2020]
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning [ICLR2020] [Tensorflow] [Pytorch]
• A General Framework for Uncertainty Estimation in Deep Learning [ICRA2020]
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation [ECCV2020] [Pytorch]
• A Simple Baseline for Bayesian Uncertainty in Deep Learning [NIPS2019] [Pytorch]
• Lightweight Probabilistic Deep Networks [CVPR2018] [Pytorch]
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning [ICML2018]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] [Tensorflow]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] [Tensorflow]
• Simple and scalable predictive uncertainty estimation using deep ensembles [NIPS2017]

Journal

• One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]
• Bayesian modeling of uncertainty in low-level vision [IJCV1990]

Arxiv

• SoftDropConnect (SDC) – Effective and Efficient Quantification of the Network Uncertainty in Deep MR Image Analysis [arxiv2022]

Conference

• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• Dropout Sampling for Robust Object Detection in Open-Set Conditions [ICRA2018]
• Concrete Dropout [NIPS2017]
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [ICML2016]

Journal

• article

Arxiv

• DEUP: Direct Epistemic Uncertainty Prediction [arxiv2020]
• Learning Confidence for Out-of-Distribution Detection in Neural Networks[arxiv2018]

Conference

• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] [Pytorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR2021Workshop]
• Learning Loss for Test-Time Augmentation [NIPS2020]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] [Pytorch]
• Addressing failure prediction by learning model confidence [NeurIPS2019][Pytorch]
• Learning loss for active learning [CVPR2019] [Pytorch] (unofficial codes)
• Structured Uncertainty Prediction Networks [CVPR2018] [Tensorflow]
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?[NIPS2017]
• Estimating the Mean and Variance of the Target Probability Distribution [(ICNN94)]

Journal

• Confidence Estimation via Auxiliary Models [TPAMI2021]

Arxiv

• Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates [arxiv2021]
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [arxiv2021]
• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures." [arxiv2021]
• Quantifying uncertainty with GAN-based priors [arxiv2019]

Conference

• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021]
• MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks [ICCV2021] [Pytorch]
• Training independent subnetworks for robust prediction [ICLR2021]
• Uncertainty-aware GAN with Adaptive Loss for Robust MRI Image Enhancement [ICCVWorkshop2021]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Uncertainty-Aware Deep Classifiers using Generative Models [AAAI2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] [Pytorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019] [Pytorch]
• On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks [NIPS2019]

Journal

• article

Arxiv

• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification [arxiv2021]
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration [arxiv2021][Pytorch]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [arxiv2020]
• Towards Understanding Label Smoothing [arxiv2020]
• An Investigation of how Label Smoothing Affects Generalization[arxiv2020]
• On Fairness and Calibration[arxiv2017]

Conference

• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• From label smoothing to label relaxation [AAAI2021]
• Calibrating Deep Neural Networks using Focal Loss [NIPS2020] [Pytorch]
• Stationary activations for uncertainty calibration in deep learning [NIPS2020]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Regularization via structural label smoothing [ICML2020]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] [Pytorch]
• Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision [CVPRW2020] [Pytorch]
• When does label smoothing help? [NIPS2019]
• Verified Uncertainty Calibration [NIPS2019]
• Generalized zero-shot learning with deep calibration network [NIPS2018]
• Measuring Calibration in Deep Learning [CVPRW2019]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
• On calibration of modern neural networks. [ICML2017]

Journal

• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018][Python]

Arxiv

• Effective Uncertainty Estimation with Evidential Models for Open-World Recognition [arxiv2022]
• A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation [arxiv2021]
• Regression Prior Networks [arxiv2020]
• Uncertainty estimation in deep learning with application to spoken language assessment[phdthesis2019]
• Inhibited softmax for uncertainty estimation in neural networks [arxiv2018].

Conference

• Improving Evidential Deep Learning via Multi-task Learning [AAAI2022]
• Evaluating robustness of predictive uncertainty estimation: Are Dirichlet-based models reliable? [ICML2021]
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [NIPS2020] [Pytorch]
• Conservative Uncertainty Estimation By Fitting Prior Networks [ICLR2020]
• Noise Contrastive Priors for Functional Uncertainty [UAI2020]
• Deep Evidential Regression [NIPS2020] [Tensorflow]
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness [NIPS2019]
• Evidential Deep Learning to Quantify Classification Uncertainty [NIPS2018] [Pytorch]
• Predictive uncertainty estimation via prior networks [NIPS2018]

Journal

• Information Aware max-norm Dirichlet networks for predictive uncertainty estimation [NeuralNetworks2021]

Arxiv

• Deep Deterministic Uncertainty: A Simple Baseline [arxiv2021] [Pytorch]
• Deep Deterministic Uncertainty for Semantic Segmentation [arxiv2021]
• On the Practicality of Deterministic Epistemic Uncertainty [arxiv2021]
• The Hidden Uncertainty in a Neural Network’s Activations [arxiv2020]
• A simple framework for uncertainty in contrastive learning [arxiv2020]
• Distance-based Confidence Score for Neural Network Classifiers [arxiv2017]

Conference

• Improving Deterministic Uncertainty Estimation in Deep Learning for Classification and Regression [CoRR2021]
• Training normalizing flows with the information bottleneck for competitive generative classification [NIPS2020]
• Simple and principled uncertainty estimation with deterministic deep learning via distance awareness [NIPS2020]
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network [ICML2020] [Pytorch]
• Single-Model Uncertainties for Deep Learning [NIPS2019] [Pytorch]
• Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation [ICCV2019] [Pytorch]

Journal

• article

Arxiv

• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors.[Arxiv2022]
• Testing for Outliers with Conformal p-values [arxiv2021] [python]

Conference

• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles [UAI2020] [Pytorch]
• Classification with Valid and Adaptive Coverage [NIPS2020]
• Conformal Prediction Under Covariate Shift [NIPS2019]
• Conformalized Quantile Regression [NIPS2019]
• Single-Model Uncertainties for Deep Learning [NIPS2019] [Pytorch]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] [Tensorflow]

Journal

• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]

Arxiv

• Deep Deterministic Uncertainty for Semantic Segmentation [arxiv2021]
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation [arxiv2018]

Conference

• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021]
• Classification with Valid and Adaptive Coverage [NIPS2020]
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation [ACCV2020]
• Human Uncertainty Makes Classification More Robust [ICCV2019]
• Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]
• Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation [ICCV2019]
• Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation [MICCAI2019][Pytorch]
• A Probabilistic U-Net for Segmentation of Ambiguous Images [NIPS2018] [Pytorch]
• Evidential Deep Learning to Quantify Classification Uncertainty [NIPS2018] [Pytorch]
• Lightweight Probabilistic Deep Networks [CVPR2018][Pytorch]
• Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding [BMVC2017]

Journal

• Explainable machine learning in image classification models: An uncertainty quantification perspective." [KnowledgeBased2022]

Arxiv

• UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed TomographarXiv [arxiv2022]
• On Monocular Depth Estimation and Uncertainty Quantification using Classification Approaches for Regression [arxiv2022]
• Efficient Gaussian Neural Processes for Regression [arxiv2021]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [arxiv2020]

Conference

• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] [Pytorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• Deep Evidential Regression [NIPS2020] [Tensorflow]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] [Pytorch]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] [Pytorch]
• Fast Uncertainty Estimation for Deep Learning Based Optical Flow [IROS2020]
• Inferring Distributions Over Depth from a Single Image [IROS2019] [Tensorflow]
• Multi-Task Learning based on Separable Formulation of Depth Estimation and its Uncertainty [CVPRW]
• Lightweight Probabilistic Deep Networks [CVPR2018][Pytorch]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] [Tensorflow]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
• Structured Uncertainty Prediction Networks [CVPR2018] [Tensorflow]

Journal

• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]
• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018][Python]

Arxiv

• Generalized out-of-distribution detection: A survey [arxiv2021]
• Towards Total Recall in Industrial Anomaly Detection [arxiv2021] [Pytorch]
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? [arxiv2021]
• Exploring the Limits of Out-of-Distribution Detection [arxiv2021]
• DATE: Detecting Anomalies in Text via Self-Supervision of Transformers [arxiv2021]
• Frequentist uncertainty estimates for deep learning [arxiv2018]

Conference

• VOS: Learning What You Don't Know by Virtual Outlier Synthesis [ICLR2022] [Pytorch]
• Anomaly Detection via Reverse Distillation from One-Class Embedding [CVPR2022]
• Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection [WACV2022] [Pytorch]
• PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization [ICPR2020] [Pytorch]
• Detecting out-of-distribution image without learning from out-of-distribution data." [CVPR2020]
• Learning Open Set Network with Discriminative Reciprocal Points [ECCV2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020][Pytorch]
• Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection [ICCV2019] [Pytorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019][Pytorch]

Journal

• One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

• MUAD: Multiple Uncertainties for Autonomous Driving benchmark for multiple uncertainty types and tasks [arxiv2022]
• The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection [IJCV2021]
• Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning [arxiv2021][Tensorflow]

• Bayesian Torch [github]
• A Bayesian Neural Network library for PyTorch [github]
• Uncertainty Toolbox [github]
• Mixture Density Networks (MDN) for distribution and uncertainty estimation [github]

• Yarin Gal: BAYESIAN DEEP LEARNING 101 [website]
• MIT 6.S191: Evidential Deep Learning and Uncertainty (2021) [Youtube]

Awesome conformal prediction [github]