PPO gets stuck in custom environment about ppo-for-beginners HOT 6 CLOSED

Thank you for your help! Yes close the issue there are no further issues!

from ppo-for-beginners.

Let me add that in stablebaselines3 I am able to complete the task. Success is increasing and episode length is decreasing which are both an indication of the model learning.

from ppo-for-beginners.

Hi Nikolas, thanks for using my implementation! I want to test your code locally (also because it's hard to tell what's wrong from just looking at it), but want to ask a few questions:

1. What are your imports for each file?
2. Where is init_weights in self.apply(init_weights) defined?
3. What values do you input to PPO constructor?

Will probably ask more along the way, but just want these clarifications for now.

from ppo-for-beginners.

Thank you for your answer Eric! To give you more information I have a custom gym environment in using pybullet for a pick and place task. The action space is discrete that denotes approach and grasp an object and finally move it to a target location so 0, 1, 2 respectively. The actual robot movements are hardcoded.

1. The import for each file are the regular imports torch, numpy etc. I am not sure why you are asking this so until you get back this is my answer!
2. It is defined in a file where the models used for the policy are defined and that initialises the weights of the policy.

def init_weights(m):
    class_name = m.__class__.__name__
    if class_name.find("Linear") != -1:
        torch.nn.init.xavier_normal_(m.weight)

3. The values are the environment on which to train the policy, the policy network, the learning rate (3e-4), betas passed to Adam optimizer (0.9, 0.999), the GAE lambda (0.95), clip range for the loss (0.2), the entropy coefficient (0.01) if I am correct you don't use this and it is easy to remove it for my case, the value loss coefficient (0.5), the max grad norm (0.5), the number of timsteps per update (2000) and the number of updates of the policy (5). Finally I pass in the tensorboard summary writer from Pytorch.

Is my implementation of PPO with GAE correct? I struggle to get a handle of how to manipulate all the data structures. I can try to run the code without GAE and simply use the advantage to see the effect if any.

from ppo-for-beginners.

@ericyangyu I remove the GAE implementation from the loss function and the implementation works. So it seems my GAE implementation had errors, at least that is my assumption. It seems to work well like this anyway!

from ppo-for-beginners.

This is great news Nikolas! Glad you were able to figure out your issue :) Apologies for the previous post, I meant to remove question 1 since question 2 was supposed to be the replacement. I'm going to go ahead and close this issue soon if there are no further issues.

from ppo-for-beginners.