hutec / uncertaintynn Goto Github PK

View Code? Open in Web Editor NEW

138.0 8.0 30.0 6.3 MB

Implementation and evaluation of different approaches to get uncertainty in neural networks

Python 0.74% Jupyter Notebook 99.26%

uncertainty machine-learning bayesian-network

uncertaintynn's Introduction

Uncertainty in Neural Networks

This project contains different implementation and evaluations of approaches to model uncertainty in neural networks.

Bootstrapping-Method from Osband et al.
(Monte Carlo)Dropout-Method by Gal
Combined method (heteroscedastic aleatoric + epistemic) from Kendal & Gal
Mixture Density Networks as used by Choi et al.

Those models are evaluated on 2D data for function approximation. Specifically there is a dataset having six points at (-1,1) , (-1,-1), (0,1), (0,-1), (1,1), (1,-1) which shows problems with the Dropout and Combined Method. And a ''x + sin(x)'' function with added noise.

Additionally the evaluation is done on MNIST data, for which I crafted adversarial attacks to evaluate the effectiveness of the Uncertainty methods.

Methods

Bootstrapping

The idea is to have a network with k distinct heads that share a network. The dataset is masked, so that every head only sees a subset of all data. Predictive variance and mean can then be gained by averaging over the prediction of every head.

https://arxiv.org/pdf/1602.04621v3.pdf

Monte-Carlo Dropout

Using dropout during training and test time which is approximate variational inference. Mean and variance is gained by doing stochastic forward passes (MC Dropout) and averaging over the outputs. This model can't differentiate between aleatoric and epistemic uncertainty.

http://mlg.eng.cam.ac.uk/yarin/thesis/thesis.pdf

Combined (Aleatoric + Epistemic Uncertainty) Method

Using Monte-Carlo Dropout and a modified loss function, you can get heteroscedastic aleatoric and epistemic uncertainty separated and also combine them.

https://arxiv.org/pdf/1703.04977.pdf

Mixture Density Networks

The last layer(s) are replaced by a layer that output Gaussian Mixture Models.

https://arxiv.org/pdf/1709.02249.pdf

Results

Results can be found in results or generated with

python evaluation/boostrap_evaluation.py
python evaluation/combined_evaluation.py
python evaluation/dropout_evaluation.py
python evaluation/mixture_evaluation.py

or all at once with python evaluation/evaluate_all.py

TODO

Working with higher dimensional data (MNIST)
Analyze influence of adversarial attacks

Problems

When running on the command-line, you might have to set $PYTHONPATH to the root dir: export PYTHONPATH=.

uncertaintynn's People

Contributors

Stargazers

Watchers

uncertaintynn's Issues

bootstrap dropout is 0 at evaluation

Hello @hutec

Great work really!!

I wanted to ask why is dropout 0 in here:

feed_dict = {x_placeholder: x_eval,
                 dropout_placeholder: 0,
                 mask_placeholder: np.ones(shape=(len(x_eval), n_heads, 1))}

This is from the following code

Shouldn't it be equal to dropout?

combined model loss

Hey,
great work @hutec.

I have a doubt though, in the paper "What Uncertainties Do We Need in Bayesian Deep
Learning for Computer Vision?", the loss function doesn't invoke the eps and T = 20 # Number of Monte Carlo Integration steps for adding noise to the uncertainty. Your loss function is not clear to me, could you please elaborate a little? or point to some resources which build this concept thoroughly and mathematically.

Thanks

About dropout rate in training

Hi. I have a question.

I run your "coombined_evaluation.py" and it works well.
There is a part I don't understand.
In "combined_evaluation" function , you put the dropout values as 0.2
-(sess, x_placeholder, dropout_placeholder = combined_training(x, y, 0.2, learning_rate, epochs))
I thought even if I train the model, I have to set the dropout value same with prediction.

Am I wrong?

Sorry to my poor english and Thank you.

Tensor not iterable in MNIST Training.ipynb

I'm trying to work through the MNIST Training notebook but am having trouble getting it to run as written. I think it may be because of a new TF version, or perhaps an uncaught error? Specifically, in cell 7:

for head in heads:
    **logits, class_prob = head  # unpack**
    loss = tf.losses.softmax_cross_entropy(onehot_labels=y_data, logits=logits)
    loss_per_head.append(loss)
    train_per_head.append(optimizer.minimize(loss))
    correct_prediction = tf.equal(tf.argmax(class_prob, 1), tf.argmax(y_data, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    accuracy_per_head.append(accuracy)

Returns the error "Tensor objects are not iterable when eager execution is not enabled". I'm using tf version 1.10.0. Anyone have tips on how to configure this to run correctly?

No batches