greydanus / baby-a3c Goto Github PK

Dear Author,

I think when an episode is done, hx should be reset. I am not sure whether it's a bug in
https://github.com/greydanus/baby-a3c/blob/master/baby-a3c.py#L144

If I disabled all GPUs then I get an error
"*** Error in `python': corrupted size vs. prev_size: 0x0000000000863430 *** "

If I don't disable GPU then I get error

terminate called after throwing an instance of 'std::runtime_error'
what(): CUDA error (3): initialization error
terminate called after throwing an instance of 'std::runtime_error'
what(): CUDA error (3): initialization error
terminate called after throwing an instance of 'std::runtime_error'
what(): CUDA error (3): initialization error
terminate called after throwing an instance of 'std::runtime_error'
what(): CUDA error (3): initialization error

Then
*** Error in `python': corrupted size vs. prev_size: 0x00000000021dc430 ***

Is there any idea to fix it?

Hi @greydanus ,

I'm wondering is there a specific reason why you've added a term which decreases the entropy in the loss function. From other implementations of A3C I've seen, a factor to increase the entropy is added instead with the factor being reduced over time. My understanding is that preserving a small amount of entropy helps by encouraging exploration.

Many Thanks,
Akmal Bakar

The shared_param.grad is synced only when it is None here https://github.com/greydanus/baby-a3c/blob/master/baby-a3c.py#L159. I am kind of confused. I think we have to sync it without the condition above. That means we have to sync it whenever the local model calculates a grad. Is it auto synced somewhere? Thank you for your time

Hi, Sam.

First of all, thank so much for this code and making it available. Having such a short implementation definitely helps understanding the algorithm.

I was hoping you can give me some intuition behind your use of sum instead of mean. In my implementation of REINFORCE, A3C, GAE, A2C I use mean and things work fine. Equations in online resources seem to suggest mean is the right approach. Other implementations, also use the mean.

Now, your implementation works very well, too! I tested it myself with ATARI games and other environments, and got rock solid results.

Can you share some insights on the use of sum instead of mean?

Again, thanks so much in advance!

Hello,
would you mind adding an open-source license to this project?

Hi, everyone. When I run in train mode, the code finished with exit code 0 quickly (within 10 s) without reporting any error. But, it runs normally when in test mode with a single process or multi-process. Is there anyone facing the same problem?

Hi, I believe we should break out of the

for step in range(args.rnn_steps):

loop when done == True. Currently, when the environment indicates that the episode is done, the loop continues to go on for a couple of steps. Not sure how the Gym environment responds to that, but new values, rewards etc keep being added to the lists and that can't be good for training.

Hi, I've run the script in training mode, and even after the training was over, if I then run it in test or render mode I was given the "no saved model" message. How do I make it save the model? And how do I render the (saved and learned) policy only once the training is over? Thanks in advance for your time!

Thanks for your great implementation.
Currently Iam trying to translate it to TF2 implementation. But I find it difficult for me to understand SharedAdam part and do not know how to implement it in TF2.
Could you kindly give me some tips?
Thank you.