In your code the KL divergence is calculated by:
KLD = torch.sum(qy * (log_qy - 1. / categorical_dim), dim=-1).mean()
I think, for the 1. / categorical_dim, it should be replaced by the torch.log(1. / categorical_dim), otherwise, it is not the KL divergence.

Hi @YongfeiYan, thanks for sharing your project with us.

I would like to know how do I modify the implementation to use Bernoulli variables. I need the network to generate codes consisting of 0s and 1s.

Thanks.

I was wondering what exactly this line in the KLD calculation does:
log_ratio = torch.log(qy * categorical_dim + 1e-20)

In the definition of the ELBO loss, the KLD should be computed between the variational distribution q(z|x) and the prior p(z). How come you did not simply use the pytorch implementation of KLD (kl_div)?

Is this temp parameter needed? Isn't temp passed in during the forward pass?

I'm using your loss function code in my project and getting negative values...

Is it normal to get negative KLD value?

Hi, what does the latent_dim mean in your code? Could it be changed to other numbers? I can understand that categorical_dim means 10 categories for 10 digits, but I'm confused about the latent_dim. Thanks!

The gumbel_softmax_sample function is logits + gumbel_sample. But it should have been F.log_softmax(logits) + gumbel_sample according to the paper. Is this not a mistake?

Hey, sorry I haven't run your code to check, but here does this reset the temperature parameter after each epoch?