Code Monkey home page Code Monkey logo

sugarl's Introduction

SUGARL

Code for NeurIPS 2023 paper Active Vision Reinforcement Learning with Limited Visual Observability, by Jinghuan Shang and Michael S. Ryoo.

We propose Sensorimotor Understanding Guided Active Reinforcement Learning (SUGARL) to solve ActiveVision-RL tasks. We also introduce Active-Gym, a convenient library that modifies existing RL environments for ActiveVision-RL, with Gymnasium-like interface.

[Paper] [Project Page] [Active-Gym]

Dependency

conda env create -f active_rl_env.yaml

We highlight Active-Gym developed by us to support Active-RL setting for many environments.

Usage

  • General format:
cd sugarl       # make sure you are under the root dir of this repo
bash ./scripts/<any_setting.sh> agent/<any_agent_valid_for_that_setting.py>
  • Reproduce our experiments:
cd sugarl       # make sure you are under the root dir of this repo
bash ./scripts/robosuite_series.sh agent/<any_agent_valid_for_that_setting.py>
bash ./scripts/atari_series.sh agent/<any_agent_valid_for_that_setting.py>
bash ./scripts/atari_series_5m.sh agent/<any_agent_valid_for_that_setting.py>
bash ./scripts/atari_wp_series.sh agent/<any_agent_valid_for_that_setting.py>
bash ./scripts/dmc_series.sh agent/<any_agent_valid_for_that_setting.py>

For example, to run SUGARL-DQN on Atari

bash ./scripts/atari_series.sh agent/dqn_atari_sugarl.py
  • Sanity checks: they run through the whole process with only a tiny amount of training to check bugs
cd sugarl       # make sure you are under the root dir of this repo
bash ./scripts/atari_test.sh agent/<any_agent_valid_for_your_test.py>
bash ./scripts/dmc_test.sh agent/<any_agent_valid_for_your_test.py>
bash ./scripts/robosuite_test.sh agent/<any_agent_valid_for_your_test.py>

All experiment scripts automatically scale all tasks to your GPUs. Please modify the gpu behavior (CUDA_VISIBLE_DEVICES=<x>) in the script if

  • you want to run jobs on certain GPUs
  • either VRAM or RAM is not sufficient for scaling all jobs

In the provided scripts, 26 Atari games are in parallel, with sequentially executing each seed. 6 DMC environments x 5 seeds are all in parallel. Please do check the available RAM and VRAM on your machine before starting.

Notes

Naming:

All agents are under agent/, with the name format <base_algorithm>_<env>_<variant>.py. Each file is an individual entry for the whole process. We support DQN, SAC, and DrQ for base algorithms.

All experiment scripts are under scripts/, with the format <env>_<setting>.sh Please ensure that the env and setting match the agent when launching jobs.

Resource requirement reference (SUGARL):

  • Atari: for each game with 100k replay buffer: ~18G RAM, <2G VRAM

  • DMC: for each task with 100k replay buffer: ~18G RAM, <3G VRAM

  • Robosuite: for each task with 100k replay buffer: ~54G RAM, 4.2G VRAM

Coding style:

We follow the coding style of clean-rl so that modifications on one agent would not affect others. This does introduce lots of redundency, but is so much easier for arranging experiments and evolving the algorithm.

Citation

Please consider cite us if you find this repo helpful.

@article{shang2023active,
    title={Active Reinforcement Learning under Limited Visual Observability},
    author={Jinghuan Shang and Michael S. Ryoo},
    journal={arXiv preprint},
    year={2023},
    eprint={2306.00975},
}

Acknowledgement

We thank the implementation of clean-rl.

sugarl's People

Contributors

elicassion avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

baiyunpeng1949

sugarl's Issues

Error when trying to create conda env

When conda create the env off of the yaml file, I get the following error:

Installing pip dependencies: \ Ran pip subprocess with arguments:
['/work/ahsia/anaconda3/envs/arl/bin/python', '-m', 'pip', 'install', '-U', '-r', '/work/ahsia/autoexperiment_dataset/2306.00975/sugarl/condaenv.38az271m.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting git+https://github.com/elicassion/active-gym.git (from -r /work/ahsia/autoexperiment_dataset/2306.00975/sugarl/condaenv.38az271m.requirements.txt (line 25))
Cloning https://github.com/elicassion/active-gym.git to /tmp/pip-req-build-3nih0pn0
Resolved https://github.com/elicassion/active-gym.git to commit 6fe4b609730873a416dab1822a07c8638f61c656
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'

Pip subprocess error:
Running command git clone --filter=blob:none --quiet https://github.com/elicassion/active-gym.git /tmp/pip-req-build-3nih0pn0
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
error in active_gym setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

failed

CondaEnvException: Pip failed

Code/Script for Robosuite

Hi,

Thank you for your work. Can you please also provide the code and scripts for reproducing your Robosuite experiments?

Thanks again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.