tyxe-bdl / tyxe Goto Github PK

Anything related to the basic MNIST tutorial.

@pgarimidi @nb2838

TyXe is recomended to be used instead of Pyro HiddenLayers, but it looks like it is pinned to an older version of Pyro. What is the development status of this library?

I'd be happy to contribute, there are some marked first issues, but before adding work I'd like to find out what the original authors have in mind with this package.

Thanks for the excellent TyXe initiative.
Currently, flipout is implemented in TyXe for linear and convolutional layers. Are you considering supporting as well RNN in the near future?
If I understood well, for linear and conv layers you monkey-patched F.linear and F.conv, but I didn't see any equivalent functions for RNNs in torch.nn.functional allowing for a similar solution. Do you have any idea on how to implement this?

Hi,

First, I want to say that the work you've done is clearly amazing.

I have playing a bit with TyXe and I am trying to convert the FC layer of a pretrained ResNet to a probabilistic layer.

I have been implementing the code shown in this paper, unfortunately I was unsuccessful. When using a homoskedastic Gaussian for the likelihood function I get this error:

ValueError: Error while computing log_prob at site 'likelihood.data':
Value is not broadcastable with batch_shape+event_shape: torch.Size([128]) vs torch.Size([128, 2]).

...

Sample Sites:                      
              net.fc.weight dist             |   2 512
                           value             |   2 512
                        log_prob             |        
                net.fc.bias dist             |   2    
                           value             |   2    
                        log_prob             |        
            likelihood.data dist             | 128   2
                           value             | 128    

My code is very minimalistic:

import torch
import torch.nn as nn
import albumentations as A

import pyro
from pyro.distributions import Normal

from torch.utils.data import DataLoader

from TyXe.tyxe.priors import IIDPrior
from TyXe.tyxe.likelihoods import HomoskedasticGaussian
from TyXe.tyxe import VariationalBNN

from utils.loader import ImgLoader

import glob

transform = A.Compose(
            [
                A.SmallestMaxSize(max_size=260),
                A.ShiftScaleRotate(shift_limit=0.05, scale_limit=0.05, rotate_limit=15, p=0.5),
                A.RandomCrop(height=224, width=224),
                A.RGBShift(r_shift_limit=15, g_shift_limit=15, b_shift_limit=15, p=0.5),
                A.RandomBrightnessContrast(p=0.5),
                A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
            ]
        )

# Prepare the dataset
data_path = "/Data/train"
list_images = glob.glob(data_path + "/*.jpg")

trainset = ImgLoader(list_images, transform=transform)

trainLoader = DataLoader(trainset,
                        batch_size = 128, 
                        shuffle=True)

# Load pre-trained model ResNet and add FC layers on top
torch.hub._validate_not_a_forked_repo=lambda a,b,c: True # Fix bug for Pytorch 1.9
NN = torch.hub.load('pytorch/vision:v0.9.0', 'resnet18', pretrained=True)
NN.to('cpu')

# Freeze the layers
for param in NN.parameters():
    param.requires_grad = False

NN.fc = nn.Linear(512,2)

ll_prior = IIDPrior(Normal(0, 1), expose_all=False, expose_modules=[NN.fc])
likelihood = HomoskedasticGaussian(len(list_images), event_dim=2, scale=1)
lr_guide = pyro.infer.autoguide.AutoLowRankMultivariateNormal
bnn = VariationalBNN(NN, ll_prior, likelihood, lr_guide)
optim = pyro.optim.Adam({"lr": 1e-3})

# fit the model
bnn.fit(trainLoader, optim, 5)

Thanks for your help already!

Dear,
I have used pip install tyxe but cannot import it. >> no such module.
How can I resolve this problem?
Thanks.

Hi !

First thank you very much for this repo, it is very helpful for people who are new to BNNs like me.

I have started to use TyXe for a convolutional BNN, starting from your resnet.py example.
After getting some unexpected behavior during training I have noticed that in the callback function the network was not set into evalutation mode. It is resulting in an important data leakage by training the network on the test dataset as well if I am correct.

I would recommend to start and finish the callback function with respectively b.eval() and b.train().

Hope this helps,
Regards!

Hi,

I am trying to use your library to turn UNet into a Bayesian Unet. I paste the code below: in the implementation UNet works as a pixel-to-pixel translator for 3D data. The code follows your regression example (as I am also doing regression but for higher dimensional data).

When I run the code I got a run-time error:
ValueError: Expected parameter scale (Tensor of shape (4, 1, 32, 32, 16)) of distribution Normal(loc: torch.Size([4, 1, 32, 32, 16]), scale: torch.Size([4, 1, 32, 32, 16])) to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values...

I expect that the problem is with a wrong selection of the prior and/or guide. I would appreciate any suggestion which will make the model to learn.

Regards,
Zbisław

The code:

from functools import partial

import torch
import torch.nn as nn
import torch.utils.data as data

import pyro
import pyro.distributions as dist

import tyxe

def double_convolution(in_channels, out_channels):
"""
In the original paper implementation, the convolution operations were
not padded but we are padding them here. This is because, we need the
output result size to be same as input size.
"""
conv_op = nn.Sequential(
nn.Conv3d(in_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.Conv3d(out_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True)
)
return conv_op

class UNet(nn.Module):
def init(self, num_classes):
super(UNet, self).init()

    self.max_pool3d = nn.MaxPool3d(kernel_size=2, stride=2)

    # contracting path
    # each convolution is applied twice
    self.down_convolution_1 = double_convolution(3, 64)
    self.down_convolution_2 = double_convolution(64, 128)
    self.down_convolution_3 = double_convolution(128, 256)
    self.down_convolution_4 = double_convolution(256, 512)
    self.down_convolution_5 = double_convolution(512, 1024)

    # expanding path
    self.up_transpose_1 = nn.ConvTranspose3d(
        in_channels=1024, out_channels=512,
        kernel_size=2, 
        stride=2)
    # below, `in_channels` again becomes 1024 as we are concatinating
    self.up_convolution_1 = double_convolution(1024, 512)
    self.up_transpose_2 = nn.ConvTranspose3d(
        in_channels=512, out_channels=256,
        kernel_size=2, 
        stride=2)
    self.up_convolution_2 = double_convolution(512, 256)
    self.up_transpose_3 = nn.ConvTranspose3d(
        in_channels=256, out_channels=128,
        kernel_size=2, 
        stride=2)
    self.up_convolution_3 = double_convolution(256, 128)
    self.up_transpose_4 = nn.ConvTranspose3d(
        in_channels=128, out_channels=64,
        kernel_size=2, 
        stride=2)
    self.up_convolution_4 = double_convolution(128, 64)

    # output => increase the `out_channels` as per the number of classes
    self.out = nn.Conv3d(
        in_channels=64, out_channels=num_classes, 
        kernel_size=1
    ) 

def forward(self, x):
    down_1 = self.down_convolution_1(x)
    down_2 = self.max_pool3d(down_1)
    down_3 = self.down_convolution_2(down_2)
    down_4 = self.max_pool3d(down_3)
    down_5 = self.down_convolution_3(down_4)
    down_6 = self.max_pool3d(down_5)
    down_7 = self.down_convolution_4(down_6)
    #down_8 = self.max_pool3d(down_7)
    #down_9 = self.down_convolution_5(down_8)        
    
    #up_1 = self.up_transpose_1(down_9)
    #x = self.up_convolution_1(torch.cat([down_7, up_1], 1))

    #up_2 = self.up_transpose_2(x)
    up_2 = self.up_transpose_2(down_7)
    x = self.up_convolution_2(torch.cat([down_5, up_2], 1))

    up_3 = self.up_transpose_3(x)
    x = self.up_convolution_3(torch.cat([down_3, up_3], 1))

    up_4 = self.up_transpose_4(x)
    x = self.up_convolution_4(torch.cat([down_1, up_4], 1))

    out = self.out(x)
    return out

################################################################################

if name == 'main':

pyro.set_rng_seed(42)

x = torch.randn((4,3,32,32,32)) * 0.5 + 0.5
y = torch.mean(x,1,keepdim=True) + torch.randn((4,1,32,32,32))*0.01

for _ in range(10):
    x2 = torch.randn((4,3,32,32,32)) * 0.5 + 0.5
    x = torch.cat([x, x2])

    y2 = torch.mean(x2,1,keepdim=True) + torch.randn((4,1,32,32,32))*0.01
    y = torch.cat([y, y2])

batchSize = 4
dataset = data.TensorDataset(x, y)
loader = data.DataLoader(dataset, batch_size=batchSize)




net = UNet(1)
prior = tyxe.priors.IIDPrior(dist.Normal(0, 1))
obs_model = tyxe.likelihoods.HeteroskedasticGaussian((4,1,32,32,32))
guide = partial(tyxe.guides.AutoNormal, init_scale=0.01)
bnn = tyxe.VariationalBNN(net, prior, obs_model, guide)


pyro.clear_param_store()
optim = pyro.optim.Adam({"lr": 1e-4})
elbos = []
def callback(bnn, i, e):
    elbos.append(e)
    
with tyxe.poutine.local_reparameterization():
    bnn.fit(loader, optim, 10000, callback)

Hi,
Recently I am doing some work in BNN for federated learning, found this nice work(TyXe) for BNN. Is there a way that TyXe can open the parameters of BNN and then we can do some aggregation among several BNNs.

For example, if we can get the BNN1 weight as Gaussian G(mean1, variance1), another BNN2 G(mean2, variance2). We can aggregate as two gaussian fusion.

really want to use and cite this code if possible.
Thanks.

Hi, thank you for making the code available, great work! I was wondering if you could add explicit license info to the repository. Thanks!

Hi, pardon, I'm writing to ask how to obtain the R^2 metric for a regression problem?

Please let us know what tutorials you think would be best for us to write. Our handles are @pgarimidi, @nb2838

Is there a way to save and load models, especially the ones using MCMC? I was hoping to use something along the lines of pyro.get_param_store but it does not work with MCMC apparently.

Anything related to the basic regression tutorial.

@pgarimidi @nb2838

Hi all,
I'm new to TyXe, but I'm experimenting an issue when I'm trying to load a (previously) trained model from the disk.

To be more precise, the returned error is as in the following:

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for VariationalBNN: Unexpected key(s) in state_dict: net_guide.rnn.weight_ih_l0.loc_unconstrained etc.

In particular, to save the model, I use a code like this:

pyro.get_param_store().save(os.path.join(output_dir, "param_store.pt"))
torch.save(model.state_dict(), os.path.join(output_dir, "best_mode.pt"))

To load the model (defined as tyxe.VariationalBNN(net, prior, likelihood, guide)) instead:

pyro.clear_param_store()
model.load_state_dict(torch.load(os.path.join(save_model_path, "best_model.pt")))
pyro.get_param_store().load(os.path.join(save_model_path, "param_store.pt"))

Where is the error?

Thank you so much.

deprecation warnings need to be addressed.

    features = torch.FloatTensor(data.features)
    labels = torch.LongTensor(data.labels)
    train_mask = torch.BoolTensor(data.train_mask)
    test_mask = torch.BoolTensor(data.test_mask)
    val_mask = torch.BoolTensor(data.val_mask)
    g = dgl.from_networkx(data.graph)

I'll make a PR after pyro upgrade

Hi，in the regression.ipynb (https://github.com/TyXe-BDL/TyXe/blob/master/notebooks/regression.ipynb),

In [11] :

plt.scatter(x, y, color="black")
plt.plot(x_test, y_test, color="black")
plt.plot(x_test, m.detach(), color="blue")
for c in range(1, 4):
plt.fill_between(x_test.squeeze(), (m - c * sd).squeeze(), (m + c * sd).squeeze(), alpha=c * 0.1, color="blue")
plt.ylim(-2, 2)

confidence interval can be got: (m - 3 * sd, m + 3 * sd).
But why? As the prediction may be not subject to Normal distribution.

Looking forward to your reply!

Best!

Hi, the above code can get mean and standard deviation for all 32 num_predictions.

Then how to get the individual value of each one?

Best regards!

Hi,

I was looking for BNN libraries related to PyTorch for my next project and I found TyXe.

However, I am unsure on whether this library is still being maintained. Any clues?

Thank you!

Thanks very much for the excellent work.
I have been using Pyro for a long time, and your work can really save a lot of people a lot of time 👍
Do you have any plan offering an example of use it to replace the normal neural networks in any reinforcement learning problems?

Firstly, thanks for the amazing repo. I'm a complete beginner and this repo really broke BNNs down simply in a very hands-on manner.

Could you guide me on how to calculate the uncertainty and break it into its epistemic & aleatoric components using TyXe?

Thanks!

Hi,
I would be really glad if I could get some help. Thank you!
I am using Tyxe: Pyro model, according to which I have converted the fc layer to probabilistic layer. I am facing the broadcastable error :

Value is not broadcastable with batch_shape+event_shape:

 net.fc.2.bias dist      |   4        
              value          |   4        
           log_prob       |            
 likelihood.data dist      |   8        
                   value       8 |   4    

I am using the categorical likelihood!

Below I have added my model for reference:

model.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
model.fc = nn.Sequential(
    nn.Linear(512, 50), 
    nn.Tanh(), 
    nn.Linear(50, 4)
)
prior = tyxe.priors.IIDPrior(dist.Normal(0, 1))
likelihood = tyxe.likelihoods.Categorical(len(training_Set),event_dim=1)
guide = tyxe.guides.AutoNormal
bnn = tyxe.VariationalBNN(model, prior, likelihood, guide)

lr = 1e-3
optimizer = pyro.optim.Adam({"lr": lr})
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
training_5epoch = []
pred_params_num_predict_1 = []

with tyxe.poutine.local_reparameterization():
    training_5epoch = bnn.fit(train_loader, optim=optimizer, num_epochs=5, callback=None, num_particles=1, closed_form_kl=True, device=device)
    pred_params_num_predict_1 = bnn.predict(test_images, num_predictions=1, aggregate=True, guide_traces=None)

Hi,

I’m trying to fit a radial BNN posterior variational approximation as per this paper.

However, since I’ll be training a BNN, I don’t want to have to write a custom guide and define this variational approximation for all of my layers, and so was trying to implement a custom AutoGuide which automatically puts a radial BNN approximation on all of my weights.

The radial approximation is defined as follows:

where I just need to sample all epsilon_MFVI from an independent standard normal distribution, normalize them, and multiply them by r, which is a scalar sampled from a standard normal.

How could I go about implementing this in TyXe?
Is there a smarter way of implementing this variational approximation?

P.S. Big fan of this project!

Thanks in advance.