wiseodd / natural-gradients Goto Github PK

I have no idea how do you compute fisher information matrix. Specifically, I don't know how do you compute p(x|θ) without using a prior in your data. Can you explain? Thanks.

In the pytorch implementation of kfac, G1_ is computed as:

G1_ = 1/m * a1.grad.t() @ a1.grad

However, the a1.grad is different from the a_1 in (1) of kfac's paper. Specifically, when you do backpropagation on the network to get a1.grad, it has a coefficient term 1/m in front of it, where m is the size of mini-batch. In other words, a1.grad = 1/m * a_1 (in kfac paper). Consequently, the G1_ is wrong. Similarly, G2_ and G3_ are also wrong.

Please correct me if I misunderstand something. Thanks!

I looked at the full_fisher numpy file and tried to change some parameters but it doesn't work. For example, when the X0 and X1 are changed as follows, the loss becomes nan after two steps:

X0 = np.random.randn(100, 2) - 2
X1 = np.random.randn(100, 2) + 2

What is the problem here? Thanks for your time.

Hi @wiseodd, thanks for great implementation! I wonder whether this can be generalized to a simple feed-forward network. In your code, the gradient wrt the parameters are computed directly. For networks, can you use auto-differentiation for that, say in pytorch? I find that is difficult because you need to get the gradient of the size (N, num_of_thetas). Since in my knowledge, when one does backpropagation, the loss should be a scalar, thus the batch N is omitted. What can one do to get the gradient for each instance in the batch?