Hi! Your repo is a very good guideline to LSTM w/ PPO. I have been trying to adapt it to my personal gym env.
My env:

• Obs: Dict {time-series-data: 60 last entries (like a window moving each timestep) of 15 features : 60x15 + 3 1d vectors}
• Actions: Box(2,)
• Aprox steps by ep: 400

What I think to use as a solution:

• LTSM (just for the time-series data) + Linear (fed with the previous result flattened + rest vectors of the obs) and then feed the actor critics policies of PPO.

Problems and doubts:

1. You have proposed to flatten the 60x15 and then use attention on that input. Can you provide a simple code example? I haven't worked with attention and there is not much info out there.
2. Could I use LTSM w Attention? or maybe dual encoder-decoder LTSM with or without Attention?. I want a simple but powerful approach, I don't think that the agent has to work on 'predicting' the next values on the time series (which as in stocks are difficult to predict), maybe it can figure out trends changes so that it must adapt to those with actions turning on or off triggers on the env. I wonder what arch can suit this better.
3. Right now I am trying to adapt your work to my env. I have an error and is that _train_mini_batch gives me this error Expected hidden[0] size (3, 1260, 128), got [3, 21, 128] when feeding the model. The issue is that recurrent_mini_batch_generator only collects states at the beginning of a seq. I don't know how to handle it as I have already made some changes in shapes that may not be correct and I feel very new to this. What do I use as batches? fixed secuences of obs? I'm messing up on how to feed the obs to the hole learning architecture. Any insight will be appreciated :)

Hi, thanks for making this nice repo! So I am interested in testing whether recurrent policy gradients can be used for image classification (sounds strange but basically each image is extremely large like 100,000 by 100,000 so it's impossible to process the entire image at once - instead I think perhaps I can train an agent to selectively navigate parts of the image sequentially, pick-up on landmarks of interests, and then make a classification).

So naturally, the goal here is not to solve a single environment (i.e. one image) since the model will just memorize the label. The goal is to train on e.g. 10,000 different environments (images), and test how well the agent + classifier will perform on held-out data. Do you have any suggestions for whether it makes sense to use your repo as a starting point, and modify it such that we either use a new environment in each mini-batch or sample from multiple environments in the same mini-batch? thanks!

Firstly, I wanted to thank you for the great project, it helped me understand Recurrent PPO better.

I mostly have one main question, regarding how the training is done, especially regarding the Actor and Critic hidden states.

In my situation, to better understand the workflow of the project, I use only 1 worker.

If I understand correctly, this worker will collect data (observations, actions, hidden states etc.) from the environment until termination. In the example of CartPole it would be something like this:
Step 1:
Observations: [-0.0058, -0.0000, -0.0079, 0.0000]
Hidden States (Actor): (tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]), tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]))

Then for Step 2 we get new Observations and Hidden States and so forth.

My question Is regarding the training phase and the actual sequence length. If in my batch I have 4 episodes (padded to have the same length of the longest episode), then (sequence_length = longest_episode_length) right? In case of course I want to be my sequence length to be the length of my episode.

Moreover, in the training, for each episode are we using only the first Hidden_State? If this is the case, then for each episode that is fed to the Networks, then the Hidden States are always going to be the initial states, like:
Hidden States (Actor): (tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]), tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]))

Hence if this is the case, what is the point of saving the hidden states during the episodes?

I might also have misunderstood this part, so I apologize in advance if this is a nonsense question :)

This issue is mainly to thank this great project.

I followed these codes, wrote a PPO algorithm with RNN, and trained it in the hallway environment of ml-agents.
I found that the parameter sequence_length has a great influence on the training results. I don’t know Why, I think sequence_length does not affect the memory length of the agent, is this correct?

The following is the reward curve, the max reward is 1:
green line: sequence_length = 8
gray line: sequence_length = 16

Hi,

I used your implementation for my research project and would like to credit the work. Is it possible to add a BibTex to the readme.md?

When I run python enjoy.py --model=./models/mortar_mayhem_grid.nn, I get this python pickle error:

pygame 2.5.1 (SDL 2.28.2, Python 3.7.16)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
  File "enjoy.py", line 71, in <module>
    main()
  File "enjoy.py", line 26, in main
    state_dict, config = pickle.load(open(model_path, "rb"))
_pickle.UnpicklingError: invalid load key, 'v'.

I followed the installation steps listed in the README for CUDA. The commands I ran were:

conda create -n recurrent-ppo python=3.7 --yes
conda activate recurrent-ppo
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
git clone https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt.git
cd recurrent-ppo-truncated-bptt
pip install -r requirements.txt
python enjoy.py --model=./models/mortar_mayhem_grid.nn

I am running these commands on Ubuntu 20.04.6.

Could you please look into this? Thanks!

Hi! Thanks for sharing the RNN-PPO implementation, it's very insightful. I have a question about the GAE calculation calc_advantages as part of the rollout buffer.

My understanding is that the GAE is calculated over a single episode at a time. However, with the current approach the GAE is calculated over all the episodes combined per worker. If I am not mistaken about the implementation of the data sampling.

The exact line I'm talking about:

I have test this repo on 'MountainCarContinuous-v0', but it could't work. What needs to be modified here to run the continuous environment?

After running the loss.backward() in line 293 of trainer.py, there emerges a problem with the "segmentation fault". Please, how to solve this problem?

Excuse me，how enjoy the model “./models/cartpole_masked.nn”？ When I run enjoy.py ， Show "RuntimeError: expected scalar type Double but found Float
"

Hi,
Why is the masked mean not used for advantage normalization? To my understand, it is padded too, no?