openai / epg Goto Github PK

View Code? Open in Web Editor NEW

246.0 16.0 56.0 468 KB

Code for the paper "Evolved Policy Gradients"

Home Page: https://arxiv.org/abs/1802.04821

License: MIT License

Python 64.52% CSS 6.12% JavaScript 20.85% HTML 8.51%

machine-learning reinforcement-learning evolutionary-strategy continuous-control meta-learning paper

epg's Introduction

Status: Archive (code is provided as-is, no updates expected)

Evolved Policy Gradients (EPG)

The paper is located at https://arxiv.org/abs/1802.04821. A demonstration video can be found at https://youtu.be/-Z-ieH6w0LA.

Houthooft, R., Chen, R. Y., Isola, P., Stadie, B. C., Wolski, F., Ho, J., Abbeel, P. (2018). Evolved Policy Gradients. arXiv preprint arXiv:1802.04821.

Installation

Install Anaconda:

curl -o /tmp/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash /tmp/miniconda.sh
conda create -n epg python=3.6.1
source activate epg

Install necessary OSX packages for MPI:

brew install open-mpi

Install necessary Python packages:

pip install mpi4py==3.0.0 scipy \
pandas tqdm joblib cloudpickle == 0.5.2 \
progressbar2 opencv-python flask >= 0.11.1 matplotlib pytest cython \
chainer pathos mujoco_py 'gym[all]'

Running

First go to the EPG code folder:

cd <path_to_EPG_folder>

Then launch the entry script:

PYTHONPATH=. python epg/launch_local.py

Experiment data is saved in <home_dir>/EPG_experiments/<month>-<day>/<experiment_name>.

Testing

First, set theta_load_path = '<path_to_theta.npy>/theta.npy' in launch_local.py according to the theta.npy obtained after running the launch_local.py script. This file should be located in /<home_dir>/EPG_experiments/<month>-<day>/<experiment_name>/thetas/.

Then run:

PYTHONPATH=. python epg/launch_local.py --test true

Visualizing experiment data

Assuming the experiment data is saved in <home_dir>/EPG_experiments/<month>-<day>/<experiment_name>, run:

PYTHONPATH=. python epg/viskit/frontend.py <home_dir>/EPG_experiments/<month>-<day>/<experiment_name>

Then go to http://0.0.0.0:5000 in your browser.

Viskit sourced from

Duan, Y., Chen, X., Houthooft, R., Schulman, J., Abbeel, P. "Benchmarking Deep Reinforcement Learning for Continuous Control". Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.

BibTeX entry

@article{Houthooft18Evolved,
author = {Houthooft, Rein and Chen, Richard Y. and Isola, Phillip and Stadie, Bradly C. and Wolski, Filip and Ho, Jonathan and Abbeel, Pieter},
title = {Evolved Policy Gradients},
journal={arXiv preprint arXiv:1802.04821},
year = {2018}}

epg's People

Contributors

Stargazers

Watchers

Forkers

dantodor hal2001 liyy3528 vvvictorlee kelvinson 4dlcreate codeaudit yifei87 lzhao4ever ml-lab ykankaya jdc08161063 amoliu ourobouros zenozhouzhao codes-kzhan fellfalla deepdrive wisbech qomposer jleni tianbingsz shubhampachori12110095 dimitry12 christopherhesse klqulei beamiter 355380o726602 daiviet01 novaintrovert afcarl ml-and-ai-repo muskanmahajan37 neotim global-localhost global19 global19-atlassian-net lunadash pgkang classicvalues flash-jaehyun andrewcyyang liuqi8827 isabella232 aniruddha123reinforcement ivaturiviswateja linnetfire ayoubjadouli w0lramd a-why-not-fork-repositories-good-luck joolstorrentecalo goompean timwee ghas-results seanpm2001 ghas-results

epg's Issues

Error while running

when I launched with code PYTHONPATH=. python epg/launch_local.py

encountered errors

/usr/bin/ld: cannot find -lGL
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/unixccompiler.py", line 196, in link
self.spawn(linker + ld_args)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "epg/launch_local.py", line 9, in
from epg.envs.random_robots import RandomHopper, DirHopper, NormalHopper
File "/home/tigerstone/Desktop/EPG_master/epg/envs/random_robots.py", line 6, in
from epg.envs.mujoco.hopper import RandomWeightHopperEnv, RandomWeightHopperDirEnv, NormalHopperEnv
File "/home/tigerstone/Desktop/EPG_master/epg/envs/mujoco/hopper.py", line 5, in
from gym.envs.mujoco import MujocoEnv, HopperEnv
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/gym/envs/mujoco/init.py", line 1, in
from gym.envs.mujoco.mujoco_env import MujocoEnv
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/gym/envs/mujoco/mujoco_env.py", line 12, in
import mujoco_py
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/init.py", line 3, in
from mujoco_py.builder import cymj, ignore_mujoco_warnings, functions, MujocoException
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 506, in
cymj = load_cython_ext(mujoco_path)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 105, in load_cython_ext
cext_so_path = builder.build()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 221, in build
built_so_file_path = self._build_impl()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 273, in _build_impl
so_file_path = super()._build_impl()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 244, in _build_impl
dist.run_commands()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/mujoco_py/builder.py", line 144, in build_extensions
build_ext.build_extensions(self)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/home/tigerstone/anaconda3/envs/epg/lib/python3.6/distutils/unixccompiler.py", line 198, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

How can I solve this?

setting mujoco environment issue

hi I installed all the requirements and tried to run your code.

but there are unresolved mujoco issues

(1)

E Exception:
E Missing path to your environment variable.
E Current values LD_LIBRARY_PATH=
E Please add following line to .bashrc:
E export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/usr/.mujoco/mujoco200/bin

after added

"export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/usr/.mujoco/mujoco200/bin"

to ~/.bashrc, I encountered different issue:

(2)

E Exception:
E Missing path to your environment variable.
E Current values LD_LIBRARY_PATH=:/home/k4ke/.mujoco/mujoco200/bin
E Please add following line to .bashrc:
E export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia-418

after added

"export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia-418"

to ~/.bashrc, I encountered different issue:

(3)

E distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

I did everything i could but they didn't wokr at all

so please how could i fix it

thank you

traj dimension error in agent.py line 142

when executing launch_local.py it throws the following error:
traj[:, obs.shape[1] + acts.shape[1]] = epg_rews
ValueError: could not broadcast input array from shape (64) into shape (512)

[FYI] Install requirement packages guide

FYI

pip install mpi4py==3.0.0 would not work.
conda config --append channels conda-forge and
conda install mpi4py==3.0.0 works for me.

pip install mujoco_py would not work.
pip install mujoco_py==2.0.2.8 works for me.

pip install 'gym[all]' would not work.
sudo apt-get install libosmesa6-dev and
conda install 'gym[all]' work for me.

Add lines below into .bashrc
export LD_LIBRARY_PATH="/home/[YOUR_USER_NAME]/.mujoco/mujoco200/bin:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/include/GL/"

Encountering "cannot find -lGL" error,
sudo apt-get install libglu1-mesa-dev works for me.

Encountering "cannot find 'patchelf'" error,
conda install -c anaconda patchelf works for me

no theta.npy

Dear:

I run PYTHONPATH=. python epg/launch_local.py, however, no theta.npy or no thetas in the /<experiemtn_name> folder.

yours

what is your gym version

it said
AttributeError: 'EnvSpec' object has no attribute 'max_episode_seconds'

in /EPG/epg/envs/random_robots.py", line 130, in meta_reset
max_episode_seconds=spec.max_episode_seconds

It may be my gym version is not the same as yours.
my gym is gym-0.14.0

Is there any plan for adding GoalAnt env?

I would like to check the out-of-distribution task learning with GoalAnt.