[QUESTION] Continously increasing loss and TD error about d3rlpy HOT 7 CLOSED

@SuryaPratapSingh37 Hi, thanks for the issue. Generally speaking, there is not usually convergence in deep RL training because of the nonstationary nature. So it is what it is. Also, the TD error is not a really good metrics. People don't usually use it to measure training progress.

from d3rlpy.

@takuseno Thanks for your reply. So could u pls guide like exactly how should I be changing the above code to make it converge? I felt cartpole is a pretty simple environment & at least the loss should have been decreasing (if not overfitting), and secondly, if not TD error what else should I use here to examine the training loss (for finding whether its overfitting or not)?

from d3rlpy.

Sadly, particularly in offline deep RL, it's very difficult to prevent divergence. So my recommendation is to give up on the convergence. Also, in offline RL, there is no good metrics to measure policy performance yet. I'd direct you to this documentation and a paper:

• https://d3rlpy.readthedocs.io/en/v2.3.0/tutorials/offline_policy_selection.html
• https://arxiv.org/abs/2007.09055

Offline deep RL still needs a lot of inventions to make it practical 😓

from d3rlpy.

Ohh.....do you know whether during training the transitions are sampled randomly from the replay buffer (if not how to randomly shuffle the transitions)?

from d3rlpy.

Yes, the mini-batch is uniformly sampled from the buffer.

from d3rlpy.

Another suggestion to prevent the divergence is using offline RL algorithms. For now, it looks like you're using DQN, which is designed for online training. If you use DiscreteCQL instead, you might get the better results in offline setting.

from d3rlpy.

Please let me close this issue since it's simply a nature of offline RL.

from d3rlpy.