SAC cannot converge to optimal policy about cleanrl HOT 3 CLOSED

Hello. Sorry for late answer.
I recall also having some difficulties getting SAC (sometimes other algorithms) to converge on a supposedly trivial task such as MountainCar.

Just from the top of my head, maybe an avenue worth exploring, encouraging more exploration with higher--alpha noise could help overcome the local optimum.

from cleanrl.

Hey, had a similar issue with discrete SAC and PPO performance. I wanted to adapt it from training on Atari to solving Minigrid. I thought there must be some issue with algorithm or environment setup but apparently I just had to spend some more time on fine-tuning parameters. Fortunately you don't have to do it by hand, as there's libraries that can handle that for you (like Optuna). I gotta admit though that SAC requires much more time to properly converge.

You could also check out Tips and tricks. Maybe that will help you to spot the issue.

Hope that helps!

from cleanrl.

Thanks a lot for your comments, @fr30!

from cleanrl.