by571 / dqn-atari-agents Goto Github PK

would you mind adding the script for making the plots?

Hello, thank you for sharing. Your work has been very helpful to me!
I encountered some issues while training in the CartPolo environment. Although the training time has accelerated, the reward continues to decrease in the later stages of training, as shown in the figure. My hyperparameters are the same as your example.
Can you tell me where the quick convergence proof is in the code? Thank you!

Hi, thanks for sharing the work. There might be an issue with the self.n_step_buffer.

When you reset the environment, self.n_step_buffer should also be reset, otherwise you will get a starting state from the previous environment and an end state from the current environment

I think this should be loss = (td_error.pow(2)*weights).mean().to(self.device) instead of loss = td_error.pow(2)*weights.mean().to(self.device). Without those brackets, loss is a vector of shape [batch_size, 1] instead of a scalar.

Command to reproduce the error:
python run_atari_dqn.py -env CartPole-v0 -agent dqn+per -frames 30000 -m 500000 --fill_buffer 50000 -eps_frames 1000 -seed 42 -info testCP

The error:

Traceback (most recent call last):
  File "run_atari_dqn.py", line 220, in <module>
    final_average100 = run(frames = args.frames//args.worker, eps_fixed=eps_fixed, eps_frames=args.eps_frames//args.worker, min_eps=args.min_eps, eval_every=args.eval_every//args.worker, eval_runs=args.eval_runs, worker=args.worker)
  File "run_atari_dqn.py", line 77, in run
    agent.step(s, a, r, ns, d, writer)
  File "/Users/user/Downloads/DQN-Atari-Agents/Agents/dqn_agent.py", line 122, in step
    loss = self.learn_per(experiences)
  File "/Users/user/Downloads/DQN-Atari-Agents/Agents/dqn_agent.py", line 239, in learn_per
    loss.backward()
  File "/Users/user/opt/miniconda3/envs/py36/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/Users/user/opt/miniconda3/envs/py36/lib/python3.6/site-packages/torch/autograd/__init__.py", line 121, in backward
    grad_tensors = _make_grads(tensors, grad_tensors)
  File "/Users/user/opt/miniconda3/envs/py36/lib/python3.6/site-packages/torch/autograd/__init__.py", line 47, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs

If you want I can submit a PR for that (although the change is a minor one:P).

@ run_atari_dqn.py
before using variable "seed", should add
seed = args.seed
run_random_policy(args.fill_buffer) takes quite a long time, so I commented that line and now it runs well

e.g. which version of pytorch you are using for this repo
thanks~

Hi. I am doing some research about the optimizer. However, I just changed the optimized used in your code from Adam to SGD but it seems that the agent doesn't learn at all.

Hello, first of all thanks for sharing your amazing job. I want to know Why I can't converge the network when I use your default setting(terminal: python run_atari_dqn.py)? the average 100 reward value always in about -20.9, which can't increase as the time-step increase. And Can you share your code about how to draw good pictures like you folder ./imgs/ pictures. Thank your very much!!