Comments (4)
Hi! Thanks for raising this issue!
I have not run this comparison before so I can't tell you why exactly you are observing such different outcomes. Generally, small implementation details can have outsized effects for RL algorithms. From a glance it eg seems that StableBaselines uses observation normalization by default while we do not. Similarly other small differences might cause the performance differences, eg in the architecture (network size, choice of normalization, ...) or the learning algorithm (eg target entropy value, multiple experience collection workers vs single worker, ...).
When I implemented SAC for this repo I verified that we can roughly match the performance of other SAC repos on a few standard OpenAI gym envs, but it is possible that other implementation choices work better on the sparser kitchen tasks you mentioned.
from spirl.
Thanks for your reply, I experimented on KitchenAllTasksV0 environment using the SAC algorithm from your project. The training process is in wanai. One of the strange phenomena is that q_target, policy_loss, and critic_loss all increase exponentially, which is not found in other SAC implementations. What could be the reason for this.
from spirl.
I am not sure why this is happening. Two things you could check:
(1) did you use Q-target clipping during training? Using this clipping can stabilize training by avoiding very large Q-errors. (you can use the existing flag clip_q_target = True
)
(2) from the WandB plots it seems that the alpha-value is increasing a lot (which can explain why the Q-values grow too) -- you could try running with a fixed alpha value instead (you would need to sweep a couple different values to find a good one that balances the reward and entropy objectives --> you can use the existing flag fixed_alpha
) -- I have sometimes found fixed alpha values to work better in the kitchen environment (maybe because it is sparse?)
from spirl.
Hi @ynulihao , may I know the code you are running to get stable-baselines3
baseline?
from spirl.
Related Issues (20)
- RuntimeError: No filenames found in /home/user_name/rl_ws/spirl/data/block_stacking HOT 4
- How to evaluate the learned embedding space and skill prior? HOT 4
- RuntimeError of gradient compuptation HOT 3
- Logger Error for --mode=val HOT 3
- parallel env HOT 3
- How to regenerate kitchen dataset? HOT 2
- Pre-trained models HOT 1
- completed_subtasks = info.pop("computed_tasks error") HOT 2
- How to get the 'kitchen-mixed-v0.py' ? HOT 7
- How can i simulate the trained policy for kitchen environment with Mujoco or another? HOT 1
- Caught RuntimeError in replica 1 on device 1. HOT 3
- Unable to download the data HOT 2
- Is it possible to change the camera zoom? HOT 1
- stacked_imgs are only available up to 2 HOT 1
- Kitchen enivronment observation is 60 instead of 30 HOT 1
- Running error "gym.error.UnregisteredEnv: No registered env with id: kitchen-mixed-v0" HOT 1
- The success rate definition of Maze Navigation Env HOT 5
- Segmentation fault (core dumped) HOT 1
- Dataset for kitchen environment HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spirl.