hongzimao / a3c Goto Github PK

I am trying to implement A3C for a particular problem.
With some tweaking, I have got it working for my specific problem.
I do have a few queries regarding the optimisation though.

1.The actor and Critic are to be optimised simultaneously right? Although it does seem like they are being optimised simultaneously, they both have totally separate models. Could you throw some light on that specific implementation?
2. In some cases of higher difficulty of my problem, the tdloss seems to have converged to 0, but the expected reward is still some way off the actual expected rewards and doesn't seem to converge completely. Do you know why that could be happening?
3. One thing I did notice in the implementation is that while training if the episode doesn't reach the done state within the train_Batch_length, it stops after that and trains it till the size with done state as false. This seems to work really good when my state space is quite small. But for Larger state spaces, it becomes really tricky to tweak this exact training batch size.

Thanks!

In the a3c.py, there is a function compute_entropy(x) which is actually not called anywhere.
Is it something that needs to be used somewhere or am I missing something?

Thanks