Comments (5)
Hi! So this is a stateful lstm implementation so the cell state is kept and sent forward through time. So the cell state at step 20 is input for the lstmcell at step 21. What is done here:
self.cx = Variable(self.cx.data)
self.hx = Variable(self.hx.data)
The hx, cx output of lstmcell the Variables are volatile and cannot be bppt so we create new Variables for the underlying data in hx, cx Variables and now they can be ready to bppt for next update.
from rl_a3c_pytorch.
Hi, thanks for your replay. Your action_train function is executed for every training step. And self.done is always False until the env resets. So you are actually setting
self.cx = Variable(self.cx.data)
self.hx = Variable(self.hx.data)
nearly every time step.
Now self.cx and self.hx are new Variables and gradients will not be passed through.
If you check the project you reference, https://github.com/ikostrikov/pytorch-a3c, it doesn't have such problem because it sets
self.cx = Variable(self.cx.data)
self.hx = Variable(self.hx.data)
every args.num_steps, instead of every step.
from rl_a3c_pytorch.
hmm your right it looks like I changed something here. I'll take a look in a little bit but very busy at the moment
from rl_a3c_pytorch.
Oh, I don't think it's the problem of GPU/CPU.
self.cx = Variable(self.cx.data)
self.hx = Variable(self.hx.data)
is ok for both GPU and CPU.
The problem is you don't want to put these two lines in the "else" condition. This will make this two lines execute every time step, except episode terminates (self.done = True).
What you want to do is to execute these 2 lines every args.num_steps (in your setting, args.num_steps = 20).
from rl_a3c_pytorch.
its fixed now should be fine now thanks!
Wow thanks for spotting had not noticed this error in repo. My version is not linked to GitHub and just been checking using trained models. And test part was fine lol. Good spot! For clarity all final performance of models posted were not trained with this bug in code. Thanks again!
from rl_a3c_pytorch.
Related Issues (20)
- Why ensure_shared_grads HOT 1
- NotImplementedError HOT 1
- Pretrained models HOT 6
- Quick question on batch processing HOT 4
- eps for Adam HOT 2
- plot rewards as a function of number of timesteps HOT 1
- Stuck when training in MsPacman-v0 HOT 7
- Reward Smoothing HOT 2
- Need for trained models HOT 2
- Cannot import test HOT 1
- Question about Test function HOT 1
- Is there any necessary to lock when update params? HOT 1
- How can I let the training automatically stop after a given number of episodes or after a given period of time? HOT 1
- Clarification needed regarding num_workers HOT 2
- question about trained models HOT 1
- UserWarning: This overload of add_ is deprecated
- Need a model, thank you HOT 1
- The links to the Gym environment evaluations is 404.
- run a3c on 8 cpus, it still slow. HOT 1
- Hyperparameters for training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rl_a3c_pytorch.