Hi!

Are predictions made available for the CIFAR-C, MedMNIST, and Diabetic Retinopathy Datasets?

Hi Pavel!

I saw that on the competition website, it says

... we will release several baselines, including stochastic MCMC, variational methods, and deep ensembles.
We will upload HMC samples on other model-dataset pairs over time.

but I can only find the HMC checkpoints on CIFAR-10, 100 with PreResNet-20 at the Colab link.

I am wondering if you can point me to the code base for the baseline methods, and other available HMC checkpoints? Thanks a lot!!

Best,
Tim

Hello Pavel!

I was trying to run the HMC checkpoints provided here, but I noticed that if I simply run the colab notebook with the full CIFAR10 test set, the test accuracy is 0.8247, which doesn't match the test accuracy provided from the ckpt_dict["accuracy"] (i.e., 0.9601807) for the default checkpoint chain 0, sample 152.

To reproduce it, you can simply change the notebook ceil from

num_inputs = 100

x_test, y_test = test_set
input, target = x_test[0, :num_inputs], y_test[0, :num_inputs]

to

x_test, y_test = test_set
input, target = x_test[0], y_test[0]

and run all.

Did I do something wrong? How can I reproduce the accuracy from ckpt_dict["accuracy"] for a give checkpoint?

Thank a lot!!