sparkmxy / my-offlinerl Goto Github PK

Since COMBO is derived from CQL, I just wonder why the auto adjustment used in CQL is not discussed in COMBO? Is it useless in COMBO?

Hi guys,

Thanks a lot for the codebase you open-sourced.

I'm trying to build on your implementation of model-based offline algorithms (mainly COMBO) to use it with other types of generative models. During my experiments I observed a numerical overflow (Losses exploding to infinity after 9-10 epochs of the agent training). Screenshot below.

This occurs in hopper-v2 environment with the d4rl medium dataset for instance.

Here are the hyperparameters:

seed = 42 

device = 'cuda'+":"+str(select_free_cuda()) if torch.cuda.is_available() else 'cpu'
obs_shape = None
act_shape = None
max_action = None

hidden_layer_size = 400
hidden_layers = 2
transition_layers = 4

transition_init_num = 7
transition_select_num = 5

real_data_ratio = 0.5

transition_batch_size = 256
policy_batch_size = 256
data_collection_per_epoch = 50e3
buffer_size = 120e4
steps_per_epoch = 1000
max_epoch = 500

learnable_alpha = True
transition_lr = 1e-3
actor_lr = 1e-4
critic_lr = 3e-4
target_entropy = None
discount = 0.99
soft_target_tau = 5e-3

num_samples = 10
learnable_beta = True
base_beta = 1.0
lagrange_thresh = 5
with_important_sampling = True

horizon = 5

Command:

python examples/train_d4rl.py --algo_name=combo --exp_name=d4rl-hopper-medium-combo-1 --task d4rl-hopper-medium-v2

Have you observed anything like this before? And if it's a matter of hyperparameters can you suggest the best hyperparameters for the hopper-medium experiment for instance?

Thank you,
Regards.