Defending against adversarial policies in YouShallNotPass by running adversarial fine-tuning. Policies are trained in an alternating fashion: after training the adversary for t₁ steps, the victim is trained for t₂ steps, then the adversary is trained again for t₃ time-steps and so on. Training times t_i increase exponentially.

Bursts training: (left) training opponents ('normal' pre-trained, adversary trained from scratch, victim policy) in an alternating way (middle) 'burst' size (right) win rate

Bursts training: (left) mean reward for agents, (right) value loss for agents

In this repository:

1. YouShallNotPass environment is exported into rllib as a multiagent environment
2. Training in 'bursts' is implemented: victim or the adversary are trained against each other, the policy trained changes every t_i time-steps, and t_i increase exponentially
3. Victim is trained against multiple adversaries as well as the normal opponent ('population-based training')
4. Stable Baselines are connected to rllib to train by sampling with rllib and optimizing with Stable Baslines

1. First, pull the image:

   $ docker pull humancompatibleai/better-adversarial-defenses

2. To run tests (will ask for a MuJoCo license)

   $ docker run -it humancompatibleai/better-adversarial-defenses

3. To run the terminal:

   $ docker run -it humancompatibleai/better-adversarial-defenses /bin/bash

1. To test the setup with rllilb PPO trainer, run:

   (adv-tf2) $ python -m ap_rllib.train --tune test

   • The script will automatically log results to Sacred and Tune

   • By-default, the script asks which configuration to run, but it can be set manually with the --tune argument.

   • Log files will appear in ~/ray_results/run_type/run_name. Use TensorBoard in this folder.,

     • run_type is determined by the configuration (config['_call']['name'] attribute). See config.py.
     • run_name is determined by tune -- see output of the train script.

   • Checkpoints will be in ~/ray_results/xxx/checkpoint_n/ where xxx and n are stored in the log files, one entry for every iteration. See an example notebook or a script obtaining the last checkpoint for details on how to do that.

   • Some specifig configurations:

     • --tune external_cartpole runs training in InvertedPendulum, using Stable Baselines PPO implementation.
       • Before running, launch the Stable Baselines server (adv-tf1) $ python -m frankenstein.stable_baselines_server
         • By-default, each policy is trained in a separate thread, so that environment data collection resumes as soon as possible
         • However, this increases the number of threads significantly in case of PBT and many parallel tune trials.
         • If the number of threads is too high, the --serial option disables multi-threaded training in Stable Baselines Server
         • The overhead is not that significant, as training finishes extremely quickly compared to data collection
     • --tune bursts_exp_withnormal_pbt_sb will run training with Stable Baselines + Bursts + Normal opponent included + PBT (multiple adversaries)

   • --verbose enables some additional output

   • --show_config only shows configuration and exits

   • --resume will re-start trials if there are already trials in the results directory with this name

     • notebook tune_pre_restart.ipynb allows to convert ray 0.8.6 checkpoints to ray 1.0.1 checkpoints

   • If you want to quickly iterate with your config (use smaller batch size and no remote workers), pass an option to the trainer

     --config_override='{"train_batch_size": 1000, "sgd_minibatch_size": 1000, "num_workers": 0, "_run_inline": 1}'

   • Large number of processes might run into the open files limit. This might help: ulimit -n 999999

2. To make a video:

   • (only on headless setups): $ Xvfb -screen 0 1024x768x24&; export DISPLAY=:0

   • Run (adv-tf2) $ python -m ap_rllib.make_video --checkpoint path/to/checkpoint/checkpoint-xxx --config your-config-at-training --display $DISPLAY

     • --steps n number of steps to make (1 is 256steps which is approximately 1 episode)
     • --load_normal True evaluate against normal opponent instead of the trained one
     • --no_video True will disable video. Use this to evaluate the performance with more episodes faster

1. We use ray because of its multi-agent support, and thus we have to use TensorFlow 2.0
2. We use stable baselines for training because we were unable to replicate results with rllib, even with an independent search for hyperparameters.
3. We checkpoint the ray trainer and restore it, and run the whole thing in a separate process to circumvent the ray memory leak issue