Comments (18)
from di-engine.
You can try gtrxl with R2D2 according to above mentioned link.
As for gtrxl with PPO, if necessary, we will add corresponding implementation. Which RL environment do you want to use gtrxl with ppo, such as some discrete action envs like lunarlander?
from di-engine.
Hmmm oddly enough, I tried that link and it was giving me some error. I will try to run it again and see what the error was. I am just trying to test it out on atari env at the moment.
from di-engine.
and I get this error
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 778, in getattr
raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'GTrXLDiscreteHead' object has no attribute 'dropout'
from di-engine.
and I get this error
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 778, in getattr raise ModuleAttributeError("'{}' object has no attribute '{}'".format( torch.nn.modules.module.ModuleAttributeError: 'GTrXLDiscreteHead' object has no attribute 'dropout'
Maybe your config is wrong, you can check it by comparing it with our atari config, i.e. for Atari env, you need to indicate 3D obs_shape
from di-engine.
oh, I see! thank you!
from di-engine.
Okay so that code works now, I didnt change the config, but it crashes after a bit. I have a RTX 3080 and 32GB of RAM.
from di-engine.
What kind of crash error you get, can you offer more details like the error traceback?
from di-engine.
I can't really see, I have everything on that docker file, it trains for a while then after several hours it exits, so I am assuming maybe it was a memory leak then sigkill was called? possibly. Were you able to run training in that environment with no issues after a very long period of time?
from di-engine.
Maybe your problem is OOM, due to the huge replay buffer of R2D2-GTrXL, you can try to rerun your experiment with smaller replay_buffer_size
. In our experiment about R2D2-GTrXL in Atari (training curve is in here), it usually needs more than 50~60GB of RAM, you can monitor the usage of RAM.
from di-engine.
I had my replay buffer at 1000, and also doesn't the replay buffer fill up fairly quickly in the beginning? or does it still fill up several hours later that would cause this OOM. Because my understanding was that if it can work for the first 3 hours, why all of a sudden does it break after that? make no sense to me.
from di-engine.
For R2D2-GTrXL, each element in replay buffer is a train_sample
, i.e., a list of transition of length unroll_len
, so it will not be full at one. I think you should monitor the usage of RAM in your experiment first, you can utilize some tools like this
from di-engine.
This is what I get after 1.6 million env_time_steps, and the mean reward is still around -20, I use the same config as what was mentioned before except I brought the experience replay buffer to 1000 vs 10,000 before. Does this look right to you?
from di-engine.
I have rerun the pong_r2d2_gtrxl.py
with buffer_size
10000 and 1000, here is the naive result:
You can see the experiment with buffer_size
1000 (blue curve) indeed shows more poor performance and until 2M env step it begins to rise. But the experiment with buffer_size
10000 exhibits the similar result with our previous experiment, so there is no bug in our implementation and you need a larger replay buffer.
BTW, the memory utilization of these two experiment are shown as follows:
buffer_size
10000, max usage 40.0GB:
buffer_size
1000, max usage 9.8GB:
from di-engine.
Okay wow, thank you for the demonstration, I didn't realize that the buffer size played such a huge role. I'm currently trying to tackle Montezuma revenge with 8 GPUs, how do I make them run r2d2gtrx on multi-GPUs with data-parallel (i'm currently doing ddp). What sort of config do you recommend? I currently am running it on 1 gpu with memory len at 256, and unroll_len 95,seq_len = 90, and not much learning is happening, I'm wondering if it's my setup or r2d2gtrxl just isn't able to solve this problem?
from di-engine.
Buffer size is of great important to off-policy value-based method, if interested, you can have a look about this paper.
For data-parallel, you can refer to this doc.
To solve Montezuma Revenge, exploration is more important than better time-series model like GTrXL, I think you can think about combine RND or Go-Explore with your current code.
from di-engine.
Okay, Because I was under the impression that games like minigrid, which this approach shouldn't have any issue solving, and Montezuma revenge are similar in that they both have sparse rewards. Maybe I am incorrect here.
Also, I tried running pong with the approach, and I still wasn't able to get your reward. It stays around -19 for 5 million time steps.
from di-engine.
@PaParaZz1 Greeting, a quick question, is there a easy way for me to concatenate a Conv2D network before the GTrXl network because it looks like I cannot directly use GTrXl for Atari environment due to obs_shape, also there is no obs_shape parameter in GTrXl.
from di-engine.
Related Issues (20)
- TD3应用混合动作空间报错,AssertionError HOT 1
- how to get the ckpt file? HOT 2
- get "TypeError: __init__() got an unexpected keyword argument 'agent_obs_shape'" when running " python3 -u smac_5m6m_masac_config.py" HOT 2
- question for SMAC HOT 3
- docker内运行lunarlander_dqn_deploy失败 HOT 6
- cannot run GTrXL demo since v0.5.0 HOT 1
- BrokenPipeError: [WinError 232] 管道正在被关闭 has occurred, when running MARL algorithm QMIX in pettingzoo
- bug when running MARL algorithm Qmix in pettingzoo HOT 3
- gym_anytrading : could not broadcast input array from shape (62,) into shape (20,3) Please help!! HOT 3
- 马里奥代码咨询 HOT 1
- No example for Trading Environments HOT 1
- KeyError: 'obs_shape' HOT 2
- How to introduce other optimizers into DI-engine? HOT 2
- RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x138 and 64x138) HOT 3
- How to define close function in custom environment HOT 3
- How to initialize semantic model in subprocess mode HOT 2
- About Replicating the PPO Performance in the Hopper-V3 Environment HOT 3
- Unexpected increase in memory overhead and Multi-GPU guidance in custom environments HOT 2
- Export multi-agent policies and shared training HOT 1
- Observation shape in the custom marl environment HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from di-engine.