Data Effiecient Rainbow with Skiing does not work about rainbow HOT 4 CLOSED

Looks like a duplicate of #41. PER can get stuck depending on the way its sampling works - as you can see from how uniform sampling works fine.

from rainbow.

Checking the paper, it seems that the algorithm was only tested on 26 games, which doesn't include "skiing", so there's no guarantee in the original work that those hyperparameters are valid for the entire Atari suite.

from rainbow.

Hi, thanks for pointing this out. However, I still do not understand why it would affect prioritized experience replay sampling. If you run the algorithm with skiing for seed=123, it simply goes into an infinite loop while sampling from the buffer. It works perfectly for uniform sampling, so I am unable to figure out why this would be the case.

from rainbow.

Ah I see, I think the fact that we use 20-step TD compounds this problem as well. Thank you so much. It makes sense now.

from rainbow.