Sixth place solution for the NeurIPS Learn to Move: Walk Around Challenge part of the conference competition track.

See my blog post for more information.

The repo master contains the code that performed best in the end: learning algorithm is Soft Actor Critic (cf paper) with automatic/learnable entropy adjustment (cf paper) and a BNAF normalizing flow (cf paper) on top of the policy; exploration algorithm is Self-supervised Exploration via Disagreement (cf paper).

The agent after 268k training steps performed like this:

I worked on several approaches concurrently, which I branched out. The branches are named after the approach implemented, e.g. modelbased implementing a 'hybrid' model-based RL, deepexpl implementing ideas from Deep Exploration using Prior Functions, nstep_returns, etc. The code in each branch is as it were. Bottom line is, code outside of the master and algorithm used above is staler (e.g. I ran various experiments with DDPG until halfway through the competition and used TD3 only at the very beginning, thus running TD3 on the final codebase may require some tweaking).

• Competition simulation platform -- opensim 4.0 and osim-rl 3.0.2 (cf http://osim-rl.stanford.edu)
• gym 0.13.1
• python 3.6
• tensorflow 1.14
• numpy 1.16
• scipy 1.3
• mpi4pi 3.0.2 (+ working MPI; only needed if running distributed DDPG)