nicklhy / memn2n Goto Github PK

The original Tensorflow code implemented by carpedm20(here) uses SGD with no weight decay and momentum to optimize the network. But it does not work properly in this MXNet code.

In details, if I set all the parameters same with the TF implementation, the loss could be very large(not nan) from the very beginning. If I use a relatively small learning rate(i.e. 0.0005) and a small gradient clip value(i.e. 5), the validation perplexity will first fall to 200-300 but then increase to 400-500 again.

I also tried adam. It can give a better validation result around 160 but still much higher than the TF implementation(114).

Currently, the only difference between these two implementations I know is the lr factor method. The TF version scales the learning rate when the loss stops to decrease. But I don't know how to implement it in MXNet. Instead, I use a normal mx.lr_scheduler.FactorScheduler.