Questions about A3C about reinforcement-learning HOT 3 CLOSED

Hi there,

The code is still work in progress (it has the same problem as #30).

Where is epsilon greedy policy? I can't find it here.

There is no epsilon-greedy in A3C. Exploration is done by adding the entropy to the loss term. You may be confusing it with the Async Q Learning from the paper?

Where is accumulation of the gradients?

That's done using the minibatching. I think calculating gradients on the minibatch should be equivalent to calculating them on single examples and adding them up. I may be wrong here though.

Every update need same forward calculation which is done in prediction and this will slow down the speed of learning. I once tried to solve this problem by using partial_run but did you solve this problem?

Yeah,partial_run is definitely one solution. I think it wasn't fully supported in previous TF versions, but I think it's now. I will try add that to make it a bit faster. However, I don't think it's a huge bottleneck. The forward computation is pretty cheap (it's only a small 3 layer CNN...) so I wouldn't expect speedup above 2x. I think the original implementation use Hogwild-style SGD updates, i.e. they don't do any locking at all. That should be faster. I'm also not sure how exactly the multithreading and CPU core assignment in Tensorflow's session works. That may be another bottleneck.

Did this code replicate the results of the paper?

Nope, not yet. I'm still trying to figure out the issue in #30 - Hopefully I can come back to this one after it's solved..

from reinforcement-learning.

Thanks for the quick answer. Yes, I confused with the Q-learning and actor-critic method. I once tried to replicate the results but eventually failed.. so I hope you can solve the problems. Anyway, I learned lots of new methods of TF from your codes and your posts. Thanks again!

from reinforcement-learning.

Thanks, same to you! I learned a lot form all the code you have on Github as well :) I'll definitely let you know if I manage to reproduce the results.

from reinforcement-learning.