About the procedure of DDPG for this implementation about deepdgp HOT 1 CLOSED

Hi lgplatero,

Thanks for the comments.
In DDPG, the actor-network tries to maximize the Q value with respect to action it takes.
Which means minimizing (- Q), i.e. negative of Q value predicted on the predicted action.
Thus, I have defined actor loss as,

self.actor_loss = -tf.reduce_mean(self.predicted_Q_on_predicted_current_action)

Here, we take the mean Q value of the batch and maximize it by minimizing (- Q_mean).
for each update, we calculate the actor gradients, which is dQ/du (partial diff of Q w.r.t. actor params) given by,

self.actor_grads = tf.gradients(self.actor_loss, self.actor.trainable_vars).

For more details, I have written a blog that you can refer; https://deshpandeshrinath.github.io/blog/2018/05/05/DDPG.

Also, I have some better results after tuning the params for half-cheetah (https://www.youtube.com/watch?v=qU8Nd9lyxlw) and post on medium about it.
(https://medium.com/@deshpandeshrinath/how-to-train-your-cheetah-with-deep-reinforcement-learning-14855518f916)

Feel free to ask questions or to suggest the feedback.
Thanks

from deepdgp.