The reward setting in Mujoco is confusing. When the agent steps a fixed distance from the starting point (i.e., 0) (2 or 20, temporarily denoted by symbol d), the agent receives a reward of 1 at each state and step. So, with this reward setup proposed by the authors, it doesn't feel like a sparse reward problem. In addition, in this reward setting proposed by the author, it feels like the agent is encouraged to go out of the circle of radius d (when stepping out of the circle of radius d, the agent can get a reward for every step even if it stands still), whereas the original dense reward setting encourages the agent to go further. So, this modification changes the original mission's intent. Finally, I tried to modify the reward, giving the agent a reward of 1 for every d distance traveled, and I found that this approach did not work.
If other readers have also read this question, please help me to answer my doubts, thank you very much!!!

Thank you so much for sharing your work,You summarize the LOGO algorithm in formula (10) in your paper, but the formula of minimize the upper bound is a minus sign，The plus sign is used in the TRPO algorithm to update parameters by maximizing, I don't know Why the same function be used to update parameters in the code，I don't see where the minus sign is？
trpo_step(policy_net, value_net, states, actions, returns, advantages, args.max_kl, args.damping, args.l2_reg)
if (kl > 6e-7):
trpo_step(policy_net, value_net_exp, states, actions, returns_exp, advantages_exp, kl, args.damping, args.l2_reg,fixed_log_probs = fixed_log_probs)

Dear author：

First of all, thank you very much for your contribution to the sparse reward problem of reinforcement learning！

Secondly, I want to know if LOGO also applies to PPO algorithm？ Because PPO algorithm has the same performance as TRPO, and it is simpler to use than TRPO.

Dear Desik Rengarajan,

Recently, I learned the LOGO algorithm you published in ICLR2022, and the method of guiding policy learning with demonstration data has achieved remarkable results in mujoco with theoretical assurance. It is a good job! I have some doubts about the collection method of behavioral data and the theoretical derivation, and hope to get your answers.

Following your instructions, I collected behavior data as follows:
Take Hopper-v2 as an example:
(1) The default iteration numbers is 1500. I trained TRPO in dense reward settings for 1000 iterations (namely, the training and test envs are all dense reward settings).
(2) Using the TRPO model trained in 1000 iterations to collecte about 10 eposides (about 3000 rows of data) as the behavioral data.

Except that the behavior data is different from the default data in LOGO code, the other training settings are consistent with the LOGO code. But my training results are very poor (as the below picture). I think it should be a problem of behavioral data collection. Can you elaborate further on your approach to constructing behavioral data, and can you open source the corresponding parts of your behavioral data collection program?

question 2:

question 3:

The reward setting in Mujoco is confusing. When the agent steps a fixed distance from the starting point (i.e., 0) (2 or 20, temporarily denoted by symbol d), the agent receives a reward of 1 at each state and step. So, with this reward setup proposed by the authors, it doesn't feel like a sparse reward problem. In addition, in this reward setting proposed by the author, it feels like the agent is encouraged to go out of the circle of radius d (when stepping out of the circle of radius d, the agent can get a reward for every step even if it stands still), whereas the original dense reward setting encourages the agent to go further. So, this modification changes the original mission's intent. Finally, I tried to modify the reward, giving the agent a reward of 1 for every d distance traveled, and I found that this approach did not work.
If other readers have also read this question, please help me to answer my doubts, thank you very much!!!