csmile-1006 / preferencetransformer Goto Github PK

Thank you for your wonderful work. Besides generating human_label with the collected indices in your file, can I collect more indices or my own indices? hope you give some useful suggestions

Hi,
I wonder if you could release the ID for each human label. It is an informative feature for analyzing the quality of the labels and its impact on policy learning. Anonymized IDs will suffice.

Thanks.

Hello, I'm having problem generating videos in the maze2d environment. When I use gym_env.render() on maze2d, it doesn't show anything. and if following your method of generating antmaze gym_env.physics.render(), there is an errorAttributeError: 'MazeEnv' object has no attribute 'physics'. any help please?

Hello, thanks for your great work. I have a question about the parameter “reverse”, in your code if "reverse" is set to be True, the data sequence will be (s, a, s, a, …), otherwise it will be (a, s, a, s, ……). And in your code you choose the later, I wonder is there any reason why you choose to put action before state? In your paper, it seems that the data sequence should be (s, a, s, a, …).

Hi,

Thanks for sharing the code!

I have a few questions regarding how to reproduce the results in the paper.

1. README says that you should run the following command to train a preference transformer model:

# Preference Transfomer (PT)
CUDA_VISIBLE_DEVICES=0 python -m JaxPref.new_preference_reward_main --use_human_label True --comment {experiment_name} --transformer.embd_dim 256 --transformer.n_layer 1 --transformer.n_head 4 --env {D4RL env name} --logging.output_dir './logs/pref_reward' --batch_size 256 --num_query {number of query} --query_len 100 --n_epochs 10000 --skip_flag 0 --seed {seed} --model_type PrefTransformer

However, I notice that this command will set config.used_weighted_sum in

Is it correct that you do not need the --transformer.use_weighted_sum flag?

2. I tried to reproduce the paper's results both with and without the --transformer.use_weighted_sum flag. Other than this flag, I strictly followed the guidelines in the README. In detail, when training the reward model, I set --num_query to 500 for *-medium-replay datasets and 100 for *-medium-expert datasets. When running IQL with the learned reward model, I set --seq_len=100, --eval_Interval=5000, --config=configs/mujoco_config.py, and --eval_episodes=10. Below are the IQL results for 8 seeds (0~7):

(Values in the parentheses denote the std of the normalized return.)

Surprisingly, I find that not using the preference attention layer actually performs better. Does this mean that the preference attention layer is not helpful? Or am I missing something? Also, do you have any ablation results regarding each component of your method?

In addition, I failed to reproduce the results on hopper-medium-replay (84.54) both with and without the preference attention layer. Can you take a look at this issue?

The problem that the paper addresses is very interesting, with the surprising difference in performance between human-labeled and synthetic data being particularly noteworthy. The release of a human-labeled dataset would undoubtedly have a significant impact on the advancement of research in this field, and I am eagerly looking forward to it.