Simple Tensorflow implementation of "Adaptive Gradient Methods with Dynamic Bound of Learning Rate" (ICLR 2019)
learning_rate
= 0.001final_lr
= 0.01beta1
= 0.9beta2
= 0.999
from AdaBound import AdaBoundOptimizer
train_op = AdaBoundOptimizer(learning_rate=0.001, final_lr=0.01, beta1=0.9, beta2=0.999, amsgrad=False).minimize(loss)
x = fully_connected(inputs=images, units=100)
x = relu(x)
logits = fully_connected(inputs=x, units=10)
Junho Kim