wellecks / vaes Goto Github PK

Hi,

In paper https://arxiv.org/pdf/1505.05770.pdf, equation 15, loss function is F(x), and we need to minimize this F(x).

The equation 15 after some transformation, first term is KL divergence (positive), senond term is reconstruct_loss(negative), third term is log_jacobian_det (negative), is my understanding correct?

So I could not understand your loss function in code, why your KL is negative and reconstruct_loss is positive?

Regards,
Wei

Hi,

The Elbo function returns monitor_functions, a dictionary of elements to monitor.

In train.py and evaluation.py, you call elbo_loss, which returns monitor_functions. So far so good. In train.py, you call optimizer.minimize(loss_op), where loss_op is the return value to the elbow function
(line 259 in train.py). minimize() should take the function to be minimized as argument.

Perhaps there is a better explanation for how the code is written since it is unlikely you could get the code to work if this is an error.

I just realized that the code calls train() and not train_simple(). The issue I mention above is in train_simple(). I assume it is an error?

Thank you.

I can train IAF and it gets c. 120 loss. The images are a little blurry but are definitely numbers.

I went back to your method of using an autoencoder for each set of flow parameters.

The structure is like this. Suppose we are using one step of IAF, so we need to generate a single mu and a single sigma that are autoregressive.

We have a neural net F operating on x to produce an intermediate hidden representation h (this is not z, NB):

h = F(x)

Then we apply another fully connected layer G to get an initial mu_0. (Sigma is the same, so I'll leave it out.)

mu_0 = G(h) = G(F(x)) = mu_0(x)

Now MADE kicks in. We use the single-layer autoencoder of MADE to transform mu_0 --> Enc(mu_0) --> mu = Dec(Enc(mu_0)). This last value is the actual flow parameter mu.

Since we used MADE to transform mu_0 to mu, we will be guaranteed that mu itself is autoregressive. Do the same for sigma.

If we need K steps of IAF, then we transform the same mu_0 using K single-layer MADEs to mu_1, ..., mu_K. Note that these mu are all decoupled from one another, and depend on each other only through their shared initial representation mu_0(x).

We should really be using convnets...
My computer can't really handle them though. I have a LeNet-style encoder training (and doing a good job!) but 20x slower than the single hidden layer encoder.

Hi,

Yet, in train(), you call:

      feed_dict[x], feed_dict[x_w] = training_data.next_batch(batch_size, whitened=False)

(my line numbers no longer correspond to yours). Note that in UnlabelledDataSet, the last few lines are:

    if whitened:
        return self.images[start:end], self._whitened_images[start:end]
    else: return self._images[start:end], self.images[start:end]

whitened is false, and self.images is not defined. Why does this work? Thanks.