lucasalegre / sumo-rl Goto Github PK

Reinforcement Learning environments for Traffic Signal Control with SUMO. Compatible with Gymnasium, PettingZoo, and popular RL libraries.

Home Page: https://lucasalegre.github.io/sumo-rl

License: MIT License

Python 100.00%

sumo rl-algorithms reinforcement-learning gym traffic-signal-control gym-env pettingzoo machine-learning deep-reinforcement-learning python

sumo-rl's Introduction

SUMO-RL

SUMO-RL provides a simple interface to instantiate Reinforcement Learning (RL) environments with SUMO for Traffic Signal Control.

Goals of this repository:

Provide a simple interface to work with Reinforcement Learning for Traffic Signal Control using SUMO
Support Multiagent RL
Compatibility with gymnasium.Env and popular RL libraries such as stable-baselines3 and RLlib
Easy customisation: state and reward definitions are easily modifiable

The main class is SumoEnvironment. If instantiated with parameter 'single-agent=True', it behaves like a regular Gymnasium Env. For multiagent environments, use env or parallel_env to instantiate a PettingZoo environment with AEC or Parallel API, respectively. TrafficSignal is responsible for retrieving information and actuating on traffic lights using TraCI API.

For more details, check the documentation online.

Install

Install SUMO latest version:

sudo add-apt-repository ppa:sumo/stable
sudo apt-get update
sudo apt-get install sumo sumo-tools sumo-doc

Don't forget to set SUMO_HOME variable (default sumo installation path is /usr/share/sumo)

echo 'export SUMO_HOME="/usr/share/sumo"' >> ~/.bashrc
source ~/.bashrc

Important: for a huge performance boost (~8x) with Libsumo, you can declare the variable:

export LIBSUMO_AS_TRACI=1

Notice that you will not be able to run with sumo-gui or with multiple simulations in parallel if this is active (more details).

Install SUMO-RL

Stable release version is available through pip

pip install sumo-rl

Alternatively, you can install using the latest (unreleased) version

git clone https://github.com/LucasAlegre/sumo-rl
cd sumo-rl
pip install -e .

MDP - Observations, Actions and Rewards

Observation

The default observation for each traffic signal agent is a vector:

    obs = [phase_one_hot, min_green, lane_1_density,...,lane_n_density, lane_1_queue,...,lane_n_queue]

phase_one_hot is a one-hot encoded vector indicating the current active green phase
min_green is a binary variable indicating whether min_green seconds have already passed in the current phase
lane_i_density is the number of vehicles in incoming lane i dividided by the total capacity of the lane
lane_i_queueis the number of queued (speed below 0.1 m/s) vehicles in incoming lane i divided by the total capacity of the lane

You can define your own observation by implementing a class that inherits from ObservationFunction and passing it to the environment constructor.

Action

The action space is discrete. Every 'delta_time' seconds, each traffic signal agent can choose the next green phase configuration.

E.g.: In the 2-way single intersection there are |A| = 4 discrete actions, corresponding to the following green phase configurations:

Important: every time a phase change occurs, the next phase is preeceded by a yellow phase lasting yellow_time seconds.

Rewards

The default reward function is the change in cumulative vehicle delay:

That is, the reward is how much the total delay (sum of the waiting times of all approaching vehicles) changed in relation to the previous time-step.

You can choose a different reward function (see the ones implemented in TrafficSignal) with the parameter reward_fn in the SumoEnvironment constructor.

It is also possible to implement your own reward function:

def my_reward_fn(traffic_signal):
    return traffic_signal.get_average_speed()

env = SumoEnvironment(..., reward_fn=my_reward_fn)

API's (Gymnasium and PettingZoo)

Gymnasium Single-Agent API

If your network only has ONE traffic light, then you can instantiate a standard Gymnasium env (see Gymnasium API):

import gymnasium as gym
import sumo_rl
env = gym.make('sumo-rl-v0',
                net_file='path_to_your_network.net.xml',
                route_file='path_to_your_routefile.rou.xml',
                out_csv_name='path_to_output.csv',
                use_gui=True,
                num_seconds=100000)
obs, info = env.reset()
done = False
while not done:
    next_obs, reward, terminated, truncated, info = env.step(env.action_space.sample())
    done = terminated or truncated

PettingZoo Multi-Agent API

For multi-agent environments, you can use the PettingZoo API (see Petting Zoo API):

import sumo_rl
env = sumo_rl.parallel_env(net_file='nets/RESCO/grid4x4/grid4x4.net.xml',
                  route_file='nets/RESCO/grid4x4/grid4x4_1.rou.xml',
                  use_gui=True,
                  num_seconds=3600)
observations = env.reset()
while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}  # this is where you would insert your policy
    observations, rewards, terminations, truncations, infos = env.step(actions)

RESCO Benchmarks

In the folder nets/RESCO you can find the network and route files from RESCO (Reinforcement Learning Benchmarks for Traffic Signal Control), which was built on top of SUMO-RL. See their paper for results.

Experiments

Check experiments for examples on how to instantiate an environment and train your RL agent.

Q-learning in a one-way single intersection:

python experiments/ql_single-intersection.py

RLlib PPO multiagent in a 4x4 grid:

python experiments/ppo_4x4grid.py

stable-baselines3 DQN in a 2-way single intersection:

Obs: you need to install stable-baselines3 with pip install "stable_baselines3[extra]>=2.0.0a9" for Gymnasium compatibility.

python experiments/dqn_2way-single-intersection.py

Plotting results:

python outputs/plot.py -f outputs/4x4grid/ppo_conn0_ep2

Citing

If you use this repository in your research, please cite:

@misc{sumorl,
    author = {Lucas N. Alegre},
    title = {{SUMO-RL}},
    year = {2019},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/LucasAlegre/sumo-rl}},
}

List of publications that use SUMO-RL (please open a pull request to add missing entries):

sumo-rl's People

Stargazers

Watchers

Forkers

ayeps coolgiserz iit-lab wenliyuan jeffrey28 danhlephuoc cbsandeep10 jeanucd yosoyisma mathematixy chickeneatrice fredtoby seudarwin rusyadisadi jarvisisfriday albenbagabaldo psychicfighters bond90 laupok321 478731807 haozhougt aswinkumar1999 yanyuema fancyerii llll-m alvinlxs liuyuqi123 zeyruca marcelomata thelastsinger aelbery kaimaoge blue-ybl huyz-git jtkhair wangjw6 sjj-2048 colouryen aflye christopherusky zeyefkey amerfathullah lucas-balbinot jungminx higherlin din1881 starqtius lisong2019 bridgeqzh vibhamasti mcgrche baihao6 raphael-arce gaurav131 angela945 joaovitorblabres xutete hunsooni jmlouw egimenez danihi kmasahiro zaheensyed christoadis redtachyon ecustboy wkkdhj grbagwe jkterry1 shansdah ankitdipto yuzhouxianzhi rshn1994 seanys asmax111 thisisjonchen vijay-jaisankar yuanzhi0515 eden1114 luckywlj aassilva rl-dlmu zhumingxu xjtsinghua bbpatil evanmath chenyanzhan shayanshafquat timchou7 ericschuma thoithoi58 x-yang1021 anjackq cswangle azuredark eveyuyi bonaldli baichuanzhou almujtaba-yaseen yvan233

sumo-rl's Issues

TraCI error: Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown

TraCI shows this error during the 6th run of training A2C & DQN 2-way single intersection examples. Also happen in DQN big intersection example during 117th run.

What is the environment configure, such as tensorflow version and so on?

How to get the waiting time & queue length by edges at every junction

Hi Lucas,

I created a road network and routes, and trained my network using sumo-rl. The output csv file from your code only gives the output of total waiting time and total vehicles stopped at each time step. However, I want to get the waiting time & queue length by edges at every junction during each time step, just like when using "--queue-output" command line from SUMO. Do I have to modify your env.py to make this happen or is there any better alternatives?

Regards,
Amer

'traci.exceptions.FatalTraCIError: connection closed by SUMO' and 'cologne8.net.xml' is not accessible'

hi, I made an error running sarsa_resco.py and sb3_grid4x4.py in the experiments and pz_test.py in the tests.

plot setting

hello, lucas!
I want to set the sumo env and rl env is: num_seconds 3600s, 500 episodes.
I have changed the setting about rou.xml and main.py.
But, I just can not change the plot.py?
Because I can not understand the code of plot.py.
I want to set the x label is episodes and y label is reward
I see that you have set the x label is seconds and y label is total waiting time.
So, can you tell me the plot.py function

ModuleNotFoundError: No module named 'numpy.testing.decorators'

Hi, I ran into this issue above when I tried: python experiments/a3c_2way-single-intersection.py . I looked up solutions online, which suggested changing the version of numpy used, tried that but the issue persists. Please help me out.

CSV files are corrupted in multiprocess environment

Hi, First of all thanks for this excellent work.

I have observed that when using the multi-process environment, different threads tried to write files at a same time and in a result the csv files are fully redundant.

The problem can be solved by using the simple environment or by setting the n_cpu argument to 1 in a vectorized environment. But is there any way to resolve the problem while keeping the same vectorized environment with multiple threads.?

Thanks

Odd error message

Hey, I'm trying to train with SB3 and sumo (which I installed via the installation instructions in the readme) and am getting the following error message:

Warning: Deprecated vehicle classes 'rail_fast' in input network.
========== LunarLanderContinuous-v2 ==========
Seed: 1705882364
Default hyperparameters for environment (ones being tuned will be overridden):
OrderedDict([('batch_size', 64),
             ('ent_coef', 0.01),
             ('gae_lambda', 0.98),
             ('gamma', 0.999),
             ('n_envs', 16),
             ('n_epochs', 4),
             ('n_steps', 1024),
             ('n_timesteps', 1000000.0),
             ('policy', 'MlpPolicy')])
Using 4 environments
Overwriting n_timesteps with n=5000000
 Retrying in 1 seconds
Traceback (most recent call last):
  File "train.py", line 288, in <module>
    model = exp_manager.setup_experiment()
  File "/home/j_k_terry/rl-baselines3-zoo/utils/exp_manager.py", line 200, in setup_experiment
    env = self.create_envs(self.n_envs, no_log=False)
  File "/home/j_k_terry/rl-baselines3-zoo/utils/exp_manager.py", line 544, in create_envs
    env = sumo_rl.ingolstadt7()
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/resco_envs.py", line 106, in ingolstadt7
    return parallel_env(**kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pettingzoo/utils/conversions.py", line 12, in par_fn
    env = env_fn(**kwargs)
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/env.py", line 29, in env
    env = SumoEnvironmentPZ(**kwargs)
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/env.py", line 290, in __init__
    self.env = SumoEnvironment(**self._kwargs)
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/env.py", line 98, in __init__
    conn) for ts in self.ts_ids}
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/env.py", line 98, in <dictcomp>
    conn) for ts in self.ts_ids}
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/traffic_signal.py", line 43, in __init__
    self.build_phases()
  File "/home/j_k_terry/sumo-rl/sumo_rl/environment/traffic_signal.py", line 85, in build_phases
    programs = self.sumo.trafficlight.getAllProgramLogics(self.id)
AttributeError: 'TrafficLightDomain' object has no attribute 'getAllProgramLogics'
Step #0.00Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown
Quitting (on error).

I googled the error and found this issue (#46), however the fix you suggested there does not appear to apply? See the following:

j_k_terry@hyperparameter-tuning-7:~/rl-baselines3-zoo/optimize_logs$ apt-cache policy sumo
sumo:
  Installed: 1.1.0+dfsg1-1
  Candidate: 1.1.0+dfsg1-1
  Version table:
 *** 1.1.0+dfsg1-1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

cannot import A3CPolicyGraph

from ray.rllib.agents.a3c.a3c_tf_policy_graph import A3CPolicyGraph
then ‘No module named 'ray.rllib.agents.a3c.a3c_tf_policy_graph'’
I can‘t find this file in the https://github.com/ray-project/ray/tree/master/rllib/agents/a3c
how can I solve this question please？

problem while running a3c_4x4grid

Hi @LucasAlegre I was trying to run training for a3c but encountered following problem..

warnings.warn(
2021-08-10 12:46:33,138	INFO services.py:1245 -- View the Ray dashboard at http://127.0.0.1:8265
/home/administrator/.local/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
2021-08-10 12:46:34,089	INFO trainer.py:706 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
2021-08-10 12:46:34,089	INFO trainer.py:718 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(pid=282439)  Retrying in 1 seconds
(pid=282440)  Retrying in 1 seconds
(pid=282439) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 9ms, vehicles TOT 0 ACT 0 BUF 0)                      
(pid=282440) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 8ms, vehicles TOT 0 ACT 0 BUF 0)                      
(pid=282440) /home/administrator/.local/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
(pid=282440)   warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
(pid=282439) /home/administrator/.local/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
(pid=282439)   warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
(pid=282440)  Retrying in 1 seconds
(pid=282440) 2021-08-10 12:46:36,975	ERROR worker.py:421 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=282440, ip=192.168.0.240)
(pid=282440)   File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
(pid=282440)   File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
(pid=282440)   File "/home/administrator/.local/lib/python3.8/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
(pid=282440)     return method(__ray_actor, *args, **kwargs)
(pid=282440)   File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 712, in __init__
(pid=282440)     "Created rollout worker with env {} ({}), policies {}".format(
(pid=282440)   File "/home/administrator/.local/lib/python3.8/site-packages/gym/core.py", line 150, in __str__
(pid=282440)     return '<{}<{}>>'.format(type(self).__name__, self.spec.id)
(pid=282440) AttributeError: 'str' object has no attribute 'id'
Traceback (most recent call last):
  File "experiments/a3c_4x4grid.py", line 30, in <module>
    trainer = A3CTrainer(env="4x4grid", config={
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/agents/trainer_template.py", line 123, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 584, in __init__
    super().__init__(config, logger_creator)
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/tune/trainable.py", line 103, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 731, in setup
    self._init(self.config, self.env_creator)
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/agents/trainer_template.py", line 147, in _init
    self.workers = self._make_workers(
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 813, in _make_workers
    return WorkerSet(
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 84, in __init__
    remote_spaces = ray.get(self.remote_workers(
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/worker.py", line 1566, in get
    raise value
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=282439, ip=192.168.0.240)
  File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
    return method(__ray_actor, *args, **kwargs)
  File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 712, in __init__
    "Created rollout worker with env {} ({}), policies {}".format(
  File "/home/administrator/.local/lib/python3.8/site-packages/gym/core.py", line 150, in __str__
    return '<{}<{}>>'.format(type(self).__name__, self.spec.id)
AttributeError: 'str' object has no attribute 'id'
(pid=282439)  Retrying in 1 seconds
(pid=282439) 2021-08-10 12:46:36,990	ERROR worker.py:421 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=282439, ip=192.168.0.240)
(pid=282439)   File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
(pid=282439)   File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
(pid=282439)   File "/home/administrator/.local/lib/python3.8/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
(pid=282439)     return method(__ray_actor, *args, **kwargs)
(pid=282439)   File "/home/administrator/.local/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 712, in __init__
(pid=282439)     "Created rollout worker with env {} ({}), policies {}".format(
(pid=282439)   File "/home/administrator/.local/lib/python3.8/site-packages/gym/core.py", line 150, in __str__
(pid=282439)     return '<{}<{}>>'.format(type(self).__name__, self.spec.id)
(pid=282439) AttributeError: 'str' object has no attribute 'id'

Can I get some help?

Thank you

How can we accout for the startup delay during phase change

Hi Lucas,

We are able to optimize the network under theoretical conditions, however during the change in the phase there will be startup delay induced .

How best can we account this delay in our reward / observation

Observation sometimes reached above 1, causing error.

I tried to train multiagent model on 4x4grid using c1 route with different algorithms. I simply edit the file a3c_4x4grid, change the route file to nets/4x4-Lucas/4x4c1.rou.xml and change the num_seconds to 20000.
However, during training, sometimes the observation reached above 1, and the trainer is terminated:

ValueError: ('Observation outside expected value range', Box(11,), array([1.        , 0.        , 1.1       , 0.05338078, 0.05338078,
       0.9510391 , 0.9510391 , 0.        , 0.        , 0.9510391 ,
       0.9510391 ]))

This only happens randomly (It seems that change the algorithm to PPO will make it easier to reproduce than the default A3C). In this case, only the third place of the observation will exceed 1, and that's the place of elapsed (self.traffic_signals[ts].time_on_phase / self.max_green).

ModuleNotFoundError: No module named 'stable_baselines3'

Hello, I am very interested in your project on reinforcement learning based traffic signal control.I am pleased to find your repository. I download the code, configure the virtual environment, and run the program. But there is an error as follows:
Traceback (most recent call last):
File "experiments/dqn_2way-single-intersection.py", line 2, in
from stable_baselines3.dqn.dqn import DQN
ModuleNotFoundError: No module named 'stable_baselines3'

I find there is not the folder named stable_baselines in the sumo-rl-master. Would you pleased to provide the stable_baselines3 directory and its contents? Thank you so much!

Making the traffic In Sync

Thanks for the wonderful project,

Tried A2C Trainer for the Traffic Simulation for reducing the waiting time, any suggestion if we need maintain the adjoining junction "In Sync"

plot prolem

After i ran ql_single-intersection.py, it generate the file "2021-05-06 13:52:39_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv", then i need to plot, follow as written in readme file:

Plotting results:
python3 outputs/plot.py -f outputs/2way-single-intersection/a3c

i ran "python3 outputs/plot.py -f outputs/2way-single-intersection/2021-05-06 13:52:39_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv" and "python3 outputs/plot.py -f outputs/2way-single-intersection/a3c",

it returns:

MS-7C98:~/project/traffic_rl/sumo-rl-master$ python3 outputs/plot.py -f outputs/single-intersection/2021-05-06 13:52:39_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv
Traceback (most recent call last):
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2898, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'total_wait_time'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "outputs/plot.py", line 87, in
ma=args.ma)
File "outputs/plot.py", line 35, in plot_df
df[yaxis] = pd.to_numeric(df[yaxis], errors='coerce') # convert NaN string to NaN value
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/core/frame.py", line 2906, in getitem
indexer = self.columns.get_loc(key)
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2900, in get_loc
raise KeyError(key) from err
KeyError: 'total_wait_time'

import ray

hello,I want torun it. But, something occurs,

ImportError: Ray must be imported before pickle5 because Ray requires a specific version of pickle5 (which is packaged along with Ray).

And, my ray is 0.8.5, pickle5 0.0.9, I add the" import pickle5" in the start. It still occurs, Import error.

Issue with ql_4x4grid.py

When I run the file I get this error: cannot import name 'Space' from 'gym.spaces'
Any solutions to this?
My gym version is 0.9.5 and I have tried other 0.8.2 as well as 0.9.4
Running in conda environment in windows.

traci error

hello,I have try to run it! And i try to run the ql_single..... ,
But, it seams like sumo have some problem.Can you help me?

ileNotFoundError: [Errno 2] No such file or directory: 'outputs/single-intersection/2021-05-28 05:29:41_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv'

Hello, I am trying to test your code but I have an error while running 'ql_single-intersection.py ':

Traceback (most recent call last):
File "experiments/ql_single-intersection.py", line 72, in
env.save_csv(out_csv, run)
File "/home/kriver91/sumo-rl/sumo_rl/environment/env.py", line 188, in save_csv
df.to_csv(out_csv_name + '_run{}'.format(run) + '.csv', index=False)
File "/home/kriver91/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/pandas/core/generic.py", line 3387, in to_csv
return DataFrameRenderer(formatter).to_csv(
File "/home/kriver91/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/pandas/io/formats/format.py", line 1083, in to_csv
csv_formatter.save()
File "/home/kriver91/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/pandas/io/formats/csvs.py", line 228, in save
with get_handle(
File "/home/kriver91/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/pandas/io/common.py", line 642, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/single-intersection/2021-05-28 05:29:41_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv'
Error: tcpip::Socket::recvAndCheck @ recv: peer shutdownT 68072 ACT 26 BUF 0)
Quitting (on error).

It seems like the code are not generating the output csv file.

Is it possible to implement multi-agent traffic single with DQN?

Hi, I saw the experiment dqn_2way-single-intersection.py, which implement by stable_baselines.deepq. Is it possible to implement multi-agent traffic single control in a 2x2 grid map by it? Sorry, I'm not familiar with stable_baselines.

And the current code sets the same signal phase in a 2x2 grid map, have you considered that signal lights in one 2x2 grid map may be different?

Thanks
Ming

FileNotFoundError: [Errno 2] No such file or directory: 'outputs/single-intersection/2021-05-06 12:36:47_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv __ and_warning: 'non-resource variables are not supported in the long term,

when ran python3 experiments/ql_single-intersection.py, i met an error:

/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:61: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Retrying in 1 seconds
/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Step #0.00 (0ms ?*RT. ?UPS, TraCI: 5ms, vehicles TOT 0 ACT 0 BUF 0)
Retrying in 1 seconds
Step #99000.00 (0ms ?*Traceback (most recent call last): 67691 ACT 29 BUF 0)
File "experiments/ql_single-intersection.py", line 72, in
env.save_csv(out_csv, run)
File "/home/project/traffic_rl/sumo-rl-master/sumo_rl/environment/env.py", line 188, in save_csv
df.to_csv(out_csv_name + '_run{}'.format(run) + '.csv', index=False)
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/core/generic.py", line 3170, in to_csv
formatter.save()
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 190, in save
compression=dict(self.compression_args, method=self.compression),
File "/home/program/anaconda3/envs/tf114/lib/python3.6/site-packages/pandas/io/common.py", line 493, in get_handle
f = open(path_or_buf, mode, encoding=encoding, errors=errors, newline="")
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/single-intersection/2021-05-06 12:36:47_alpha0.1_gamma0.99_eps0.05_decay1.0_run1.csv'
Error: tcpip::Socket::recvAndCheck @ recv: peer shutdownT 68472 ACT 30 BUF 0)
Quitting (on error).

Questions for SB3 Script

(new thread to simply the discussion of the hanging issues in #45)

"When using LIBSUMO, it is not possible to instantiate more than 1 simulation at the same time. This means that eval_callback can't work. If you use TRACI, I can easily implement multi-client support and this would be possible. But remember that TRACI is way slower than LIBSUMO, so I'm not sure whether this is advantageous."
-Could you please elaborate a bit more on all this? Using eval callback is essential to what I'm planning to do with this, and running multiple environments at once is generally desirable when learning with PPO

Regarding how rendering doesn't work in my sb3 code snippet, is there code somewhere that generates videos of the simulation rendering that I could adapt into that snippet?

Thanks again for all your help, I really appreciate it :)

Error while running any of the experiments

I faced this issue while running any of the python executables in the experiments folder

Traceback (most recent call last):
File "a3c_4x4grid.py", line 16, in
from sumo_rl import SumoEnvironment
File "/Users/vish/College/Semester 4/Mini Project/env/lib/python3.8/site-packages/sumo_rl/init.py", line 1, in
from sumo_rl.environment.env import SumoEnvironment
ModuleNotFoundError: No module named 'sumo_rl.environment'

TraCIException: Connection '0' is already active

Hello, I encounter a problem with the dqn_two_way_single_intersection.py file. When it run to the last second, the error happens, as the picture below.

Could you help me please? Thank you in advance!

Feature Request: Gym API test in CI

Gym added an API test for environments, you might want to add that to CI in addition to the existing PettingZoo API test

Multi agent seting

Hello, Lucas. I have read your code about a3c_4x4.py. Now, I have a question, you have set only one policy about policy 0.

Is there 16 agent use the same one policy? Right?

How do the ray multi agent map the taffic light id？ Can you teach me? THANK YOU

missing params in save_csv()

python3 experiments/ql_single-intersection.py
Retrying in 1 seconds
Retrying in 1 seconds?UPS, TraCI: 2ms, vehicles TOT 0 ACT 0 BUF 0)
Step #20000.00 (0ms ?*RT. ?UPS, Traceback (most recent call last): 42 BUF 0)
File "experiments/ql_single-intersection.py", line 89, in
env.save_csv()

TypeError: save_csv() missing 2 required positional arguments: 'out_csv_name' and 'run'

Step #20005.00Error: tcpip::Socket::recvAndCheck @ recv: peer shutdownBUF 0)
Quitting (on error).

the function definition is

    def save_csv(self, out_csv_name, run):
        if out_csv_name is not None:
            df = pd.DataFrame(self.metrics)
            df.to_csv(out_csv_name + '_run{}'.format(run) + '.csv', index=False)

and it's being called as only save_csv() with no arguments

Blank csv output with fixed lights

Hi!

We tried running your 'ql_2way-single-intersection.py' experiment, but when we tried setting fixed to true it runs, but the output is blank.

How do we get the output of the fixed lights? We want to use it to compare with trained model

sumocfg setting

hello,Lucas! I have run your code about the sumo settings. But I found the traffic flow was one-way. So, could you tell me ..?

How to evaluate the outcomes?

To avoid the error made by traci when training, I used libsumo and it is ok. But after it was trained, I want to evaluate the outcome and watch the viusalization with sumo-gui. So I save the model in the dqn-2way-single-intersection.py and create another py file, and import evaluate_policy from SB3 and use traci to run it. But the same problem as my last question (TraCIException: Connection '0' is already active) happened again. Therefore, I want to how to sovle it and the evaluate code is displayed as below. Thank you very much!

Disabling sumo warnings

Right now, the log file where I pipe the results of my learning code to to watch is is full of countless sumo warnings, to the point where I can't really read things and see whats going on, like this:

How can I disable it?

Env of multi intersections has a bug

There is a bug in the env of multi intersections. For example, in experiments/ql_4x4grid.py,

env = SumoEnvironment(net_file='nets/4x4-Lucas/4x4.net.xml',
                          route_file='nets/4x4-Lucas/4x4c1c2c1c2.rou.xml',
                          use_gui=True,
                          num_seconds=80000,
                          max_depart_delay=0)

there are 16 intersections but the single_agent in class SumoEnvironment(MultiAgentEnv): is Flase

    def __init__(self, net_file, route_file, out_csv_name=None, use_gui=False, num_seconds=20000, max_depart_delay=100000, time_to_teleport=-1, delta_time=5, yellow_time=2, min_green=5, max_green=50, single_agent=False):

nothing has been changed, so the state returned by the reset function maybe wrong.

Be careful to use multi intersections environments.

Saving output for ql_4x4grid.py

I tried to add a line
env.save_csv(out_csv, run)

to the python file for 4x4 grids but am getting the following error
shankrith@Aurelius:~/Traffic_Constable/sumo-rl$ python3 experiments/ql_4x4grid.py File "experiments/ql_4x4grid.py", line 49 env.save_csv(out_csv, run) ^ TabError: inconsistent use of tabs and spaces in indentation

Will the env_save command work for 4x4 grids?

Issue with a3c_4x4 grid

I have been trying to run the RLlib A3C multiagent in a 4x4 grid file (experiments/a3c_4x4grid.py) as mentioned in the documentation but every-time, I get this error. Can you please help me solve the issue? @LucasAlegre

Instructions for updating:
non-resource variables are not supported in the long term
2020-10-28 07:10:14,677	INFO services.py:1166 -- View the Ray dashboard at http://127.0.0.1:8265
/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
2020-10-28 07:10:15,750	INFO trainer.py:591 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
2020-10-28 07:10:15,750	INFO trainer.py:618 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
 Retrying in 1 seconds
Step #0.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 0 ACT 0 BUF 0)                      
(pid=19282) WARNING:tensorflow:From /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:61: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
(pid=19282) Instructions for updating:
(pid=19282) non-resource variables are not supported in the long term
(pid=19285) WARNING:tensorflow:From /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:61: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
(pid=19285) Instructions for updating:
(pid=19285) non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py:286: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
(pid=19285)  Retrying in 1 seconds
(pid=19282)  Retrying in 1 seconds
 Retrying in 1 seconds
2020-10-28 07:10:17,454	WARNING util.py:39 -- Install gputil for GPU system monitoring.
(pid=19285) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 0 ACT 0 BUF 0)                      
(pid=19285) /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
(pid=19285)   warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
(pid=19282) /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
(pid=19282)   warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
(pid=19282) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 0 ACT 0 BUF 0)                      
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 347, in run
    raise e
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 344, in run
    self._run()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 369, in _run
    item = next(rollout_provider)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 604, in _env_runner
    perf_stats=perf_stats,
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 798, in _process_observations
    policy_id).transform(raw_obs)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 1486, in _get_or_raise
    "in policy map keys {}.".format(policy_id, mapping.keys()))
ValueError: Could not find policy for agent: agent policy id '0' not in policy map keys dict_keys(['default_policy']).

(pid=19285) WARNING:tensorflow:From /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py:286: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
(pid=19285) Instructions for updating:
(pid=19285) Use tf.where in 2.0, which has the same broadcast rule as np.where
(pid=19282) WARNING:tensorflow:From /home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py:286: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
(pid=19282) Instructions for updating:
(pid=19282) Use tf.where in 2.0, which has the same broadcast rule as np.where
(pid=19285)  Retrying in 1 seconds
(pid=19282)  Retrying in 1 seconds
Traceback (most recent call last):
  File "a3c_4x4grid.py", line 47, in <module>
    print(trainer.train())  # distributed training step
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 516, in train
    raise e
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 505, in train
    result = Trainable.train(self)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/tune/trainable.py", line 336, in train
    result = self.step()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 134, in step
    res = next(self.train_exec_impl)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 756, in __next__
    return next(self.built_iterator)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 843, in apply_filter
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 843, in apply_filter
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 828, in add_wait_hooks
    item = next(it)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 551, in base_iterator
    batch = ray.get(obj_ref)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/worker.py", line 1428, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ValueError): ray::RolloutWorker.par_iter_next_batch() (pid=19285, ip=10.14.31.64)
  File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 438, in ray._raylet.execute_task.function_executor
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 1158, in par_iter_next_batch
    batch.append(self.par_iter_next())
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 1152, in par_iter_next
    return next(self.local_it)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
    for item in it:
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 288, in gen_rollouts
    yield self.sample()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 579, in sample
    batches = [self.input_reader.next()]
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 93, in next
    batches = [self.get_data()]
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 383, in get_data
    raise rollout
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 347, in run
    raise e
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 344, in run
    self._run()
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 369, in _run
    item = next(rollout_provider)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 604, in _env_runner
    perf_stats=perf_stats,
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 798, in _process_observations
    policy_id).transform(raw_obs)
  File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 1486, in _get_or_raise
    "in policy map keys {}.".format(policy_id, mapping.keys()))
ValueError: Could not find policy for agent: agent policy id `0` not in policy map keys dict_keys(['default_policy']).
(pid=19285) Exception in thread Thread-1:
(pid=19285) Traceback (most recent call last):
(pid=19285)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=19285)     self.run()
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 347, in run
(pid=19285)     raise e
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 344, in run
(pid=19285)     self._run()
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 369, in _run
(pid=19285)     item = next(rollout_provider)
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 604, in _env_runner
(pid=19285)     perf_stats=perf_stats,
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 798, in _process_observations
(pid=19285)     policy_id).transform(raw_obs)
(pid=19285)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 1486, in _get_or_raise
(pid=19285)     "in policy map keys {}.".format(policy_id, mapping.keys()))
(pid=19285) ValueError: Could not find policy for agent: agent policy id `0` not in policy map keys dict_keys(['default_policy']).
(pid=19285) 
(pid=19282) Exception in thread Thread-1:
(pid=19282) Traceback (most recent call last):
(pid=19282)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=19282)     self.run()
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 347, in run
(pid=19282)     raise e
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 344, in run
(pid=19282)     self._run()
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 369, in _run
(pid=19282)     item = next(rollout_provider)
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 604, in _env_runner
(pid=19282)     perf_stats=perf_stats,
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 798, in _process_observations
(pid=19282)     policy_id).transform(raw_obs)
(pid=19282)   File "/home/rstudio/traffic_optimisation/sumo-rl/experiments/sumo/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 1486, in _get_or_raise
(pid=19282)     "in policy map keys {}.".format(policy_id, mapping.keys()))
(pid=19282) ValueError: Could not find policy for agent: agent policy id `0` not in policy map keys dict_keys(['default_policy']).
(pid=19282) 
Error: tcpip::Socket::recvAndCheck @ recv: peer shutdownms, vehicles TOT 272 ACT 260 BUF 0
Quitting (on error)```

Issue with ql_4x4grid

Hi. When I test your experiment, I get something problem.
When I running the code with this environment, I get

return self.radix_encode([phase, elapsed] + density_queue)
  File "/home/simsimi/桌面/sumo-rl-master/sumo_rl/environment/env.py", line 286, in radix_encode
    res = res * self.radix_factors[i] + values[i]
IndexError: list index out of range

Could you help me to solve this problem? Thanks.

ModuleNotFoundError: No module named 'tensorflow.contrib'

tensorflow.contrib no longer exist in 2.0

Execution error in a3c_4x4grid.py

Hey,

When I run a3c_4x4grid.py, I get the error "AttributeError: 'TrafficLightDomain' object has no attribute 'getAllProgramLogics'". Can you please tell me how to make it work?
When this error is fixed, I would like to run a map brought from OpenStreetMap. Will it work if I just rewrite the netfile?

The training process may not work if I remove --max-depart-delay setting in sumo

When I run a3c_2way-single-intersection.py, if I remove the line '--max-depart-delay', str(self.max_depart_delay), in env.py, the total waiting times are still around 3000 at step 10000 that is inconsistent with the results in this repository. I wonder if it means that the training process does not work for the scenario.

My plotted results:

AttributeError: 'Namespace' object has no attribute 'reward'

error when running ql_single-intersection.py

Error while running 4x4 grid experiement

Am trying to run the 4x4 grid experiment but it throws the below log


python3 experiments/a3c_4x4grid.py
2019-03-29 03:35:14,273 INFO node.py:423 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-03-29_03-35-14_19093/logs.
2019-03-29 03:35:14,379 INFO services.py:363 -- Waiting for redis server at 127.0.0.1:12017 to respond...
2019-03-29 03:35:14,505 INFO services.py:363 -- Waiting for redis server at 127.0.0.1:38753 to respond...
2019-03-29 03:35:14,510 INFO services.py:760 -- Starting Redis shard with 2.07 GB max memory.
2019-03-29 03:35:14,543 INFO services.py:1384 -- Starting the Plasma object store with 3.1 GB memory using /dev/shm.
 Retrying in 1 seconds
2019-03-29 03:35:15,693UINFO policy_evaluator.py:278 -- Creating policy evaluation worker 0 on CPU (please ignore any CUDA init errors)
2019-03-29 03:35:15.694604: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-03-29 03:35:15.708626: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
2019-03-29 03:35:15.708881: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x1c98b90 executing computations on platform Host. Devices:
2019-03-29 03:35:15.708961: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.random.categorical instead.
WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
 Retrying in 1 seconds
2019-03-29 03:35:16,281 WARNING worker.py:330 -- WARNING: Falling back to serializing objects of type <class 'numpy.dtype'> by using pickle. This may be inefficient.
2019-03-29 03:35:16,282 WARNING worker.py:330 -- WARNING: Falling back to serializing objects of type <class 'mtrand.RandomState'> by using pickle. This may be inefficient.
Step #116.00 (1ms ~= 1000.00*RT, ~296000.00UPS, TraCException in thread Thread-2:
Traceback (most recent call last):
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 47, in get
    self.feed_dict, os.environ.get("TF_TIMELINE_DIR"))
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 85, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (16, 11) for Tensor '0/Placeholder:0', which has shape '(?, 6)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 148, in run
    raise e
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 145, in run
    self._run()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 167, in _run
    item = next(rollout_provider)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 316, in _env_runner
    active_episodes)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 515, in _do_policy_eval
    eval_results[k] = builder.get(v)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 50, in get
    self.fetches, self.feed_dict))
ValueError: Error fetching: [<tf.Tensor '0/Squeeze:0' shape=(?,) dtype=int64>, {'action_prob': <tf.Tensor '0/Exp_1:0' shape=(?,) dtype=float32>, 'vf_preds': <tf.Tensor '0/Reshape:0' shape=(?,) dtype=float32>}], feed_dict={<tf.Tensor '0/Placeholder:0' shape=(?, 6) dtype=float32>: [array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.9510391 , 0.9510391 , 0.95927297, 0.95927297, 0.9510391 ,
       0.9510391 ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])], <tf.Tensor '0/action:0' shape=(?,) dtype=int64>: [array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0)], <tf.Tensor '0/prev_reward:0' shape=(?,) dtype=float32>: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], <tf.Tensor '0/PlaceholderWithDefault:0' shape=() dtype=bool>: False}

(pid=19115)  Retrying in 1 seconds
(pid=19117)  Retrying in 1 seconds
(pid=19115) 2019-03-29 03:35:19,414     INFO policy_evaluator.py:278 -- Creating policy evaluation worker 2 on CPU (please ignore any CUDA init errors)
(pid=19115) 2019-03-29 03:35:19.416187: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
(pid=19115) 2019-03-29 03:35:19.421268: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
(pid=19115) 2019-03-29 03:35:19.421394: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x1aa5d30 executing computations on platform Host. Devices:
(pid=19115) 2019-03-29 03:35:19.421407: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=19115) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 0 ACT 0 BUF 0)           
(pid=19117) 2019-03-29 03:35:19,410     INFO policy_evaluator.py:278 -- Creating policy evaluation worker 1 on CPU (please ignore any CUDA init errors)
(pid=19117) 2019-03-29 03:35:19.413480: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
(pid=19117) 2019-03-29 03:35:19.418717: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
(pid=19117) 2019-03-29 03:35:19.418926: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2712d30 executing computations on platform Host. Devices:
(pid=19117) 2019-03-29 03:35:19.418944: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=19117) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
(pid=19117) Instructions for updating:
(pid=19117) Colocations handled automatically by placer.
(pid=19117) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 2ms, vehicles TOT 0 ACT 0 BUF 0)           
(pid=19115) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
(pid=19115) Instructions for updating:
(pid=19115) Colocations handled automatically by placer.
(pid=19115) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
(pid=19115) Instructions for updating:
(pid=19115) Use tf.random.categorical instead.
(pid=19117) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
(pid=19117) Instructions for updating:
(pid=19117) Use tf.random.categorical instead.
(pid=19117) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=19117) Instructions for updating:
(pid=19117) Use tf.cast instead.
(pid=19115) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=19115) Instructions for updating:
(pid=19115) Use tf.cast instead.
(pid=19115) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=19115) Instructions for updating:
(pid=19115) Deprecated in favor of operator or tf.math.divide.
(pid=19117) WARNING:tensorflow:From /home/fady/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=19117) Instructions for updating:
(pid=19117) Deprecated in favor of operator or tf.math.divide.
(pid=19115)  Retrying in 1 seconds
(pid=19117)  Retrying in 1 seconds
(pid=19115) Step #0.00 (1ms ~= 1000.00*RT, ~8000.00UPS, TraCI: 11ms, vehicles TOT 8 ACT 8 
(pid=19115) Step #1.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 8 ACT 8 BUF 0)           
(pid=19115) Step #2.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 8 ACT 8 BUF 0)           
(pid=19115) Step #3.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19115) Step #4.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19115) Step #5.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19115) Step #6.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 24 ACT 24 BUF 0)         
(pid=19115) Step #7.00 (1ms ~= 1000.00*RT, ~24000.00UPS, TraCI: 0ms, vehicles TOT 24 ACT 2
(pid=19115) Step #8.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 24 ACT 24 BUF 0)         
(pid=19115) Step #9.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 32 ACT 32 BUF 0)         
(pid=19115) Step #10.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 32 ACT 32 BUF 0)        
(pid=19115) Step #11.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 32 ACT 32 BUF 0)        
(pid=19115) Step #12.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 40 ACT 40 BUF 0)        
(pid=19115) Step #13.00 (1ms ~= 1000.00*RT, ~40000.00UPS, TraCI: 0ms, vehicles TOT 40 ACT 
(pid=19115) Step #14.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 40 ACT 40 BUF 0)        
(pid=19115) Step #15.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 48 ACT 48 BUF 0)        
(pid=19115) Step #16.00 (1ms ~= 1000.00*RT, ~48000.00UPS, TraCI: 0ms, vehicles TOT 48 ACT 
(pid=19115) Step #17.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 48 ACT 48 BUF 0)        
(pid=19115) Step #18.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 56 ACT 56 BUF 0)        
(pid=19115) Step #19.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 56 ACT 56 BUF 0)        
(pid=19115) Step #20.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 56 ACT 56 BUF 0)        
(pid=19115) Step #21.00 (1ms ~= 1000.00*RT, ~64000.00UPS, TraCI: 0ms, vehicles TOT 64 ACT 
(pid=19115) Step #22.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 64 ACT 64 BUF 0)        
(pid=19115) Step #23.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 64 ACT 64 BUF 0)        
(pid=19115) Step #24.00 (1ms ~= 1000.00*RT, ~72000.00UPS, TraCI: 0ms, vehicles TOT 72 ACT 
(pid=19115) Step #25.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 72 ACT 72 BUF 0)        
(pid=19115) Step #26.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 72 ACT 72 BUF 0)        
(pid=19115) Step #27.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 80 ACT 80 BUF 0)        
(pid=19115) Step #28.00 (1ms ~= 1000.00*RT, ~80000.00UPS, TraCI: 0ms, vehicles TOT 80 ACT 
(pid=19115) Step #29.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 80 ACT 80 BUF 0)        
(pid=19115) Step #30.00 (1ms ~= 1000.00*RT, ~88000.00UPS, TraCI: 0ms, vehicles TOT 88 ACT 
(pid=19115) Step #31.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 88 ACT 88 BUF 0)        
(pid=19115) Step #32.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 88 ACT 88 BUF 0)        
(pid=19115) Step #33.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 96 ACT 96 BUF 0)        
(pid=19115) Step #34.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 96 ACT 96 BUF 0)        
(pid=19115) Step #35.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 96 ACT 96 BUF 0)        
(pid=19115) Step #36.00 (1ms ~= 1000.00*RT, ~104000.00UPS, TraCI: 1ms, vehicles TOT 104 AC
(pid=19115) Step #37.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 104 ACT 104 BUF 0)      
(pid=19115) Step #38.00 (1ms ~= 1000.00*RT, ~104000.00UPS, TraCI: 0ms, vehicles TOT 104 AC
(pid=19115) Step #39.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 112 ACT 112 BUF 0)      
(pid=19115) Step #40.00 (1ms ~= 1000.00*RT, ~112000.00UPS, TraCI: 0ms, vehicles TOT 112 AC
(pid=19115) Step #41.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 112 ACT 112 BUF 0)      
(pid=19115) Step #42.00 (4ms ~= 250.00*RT, ~30000.00UPS, TraCI: 0ms, vehicles TOT 120 ACT 
(pid=19115) Step #43.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 120 ACT 120 BUF 0)      
(pid=19115) Step #44.00 (1ms ~= 1000.00*RT, ~120000.00UPS, TraCI: 0ms, vehicles TOT 120 AC
(pid=19115) Step #45.00 (1ms ~= 1000.00*RT, ~128000.00UPS, TraCI: 0ms, vehicles TOT 128 AC
(pid=19115) Step #46.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 128 ACT 128 BUF 0)      
(pid=19115) Step #47.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 128 ACT 128 BUF 0)      
(pid=19115) Step #48.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 136 ACT 136 BUF 0)      
(pid=19115) Step #49.00 (1ms ~= 1000.00*RT, ~136000.00UPS, TraCI: 0ms, vehicles TOT 136 AC
(pid=19115) Step #50.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 136 ACT 136 BUF 0)      
(pid=19115) Step #51.00 (1ms ~= 1000.00*RT, ~144000.00UPS, TraCI: 0ms, vehicles
(pid=19117) Step #0.00 (1ms ~= 1000.00*RT, ~8000.00UPS, TraCI: 5ms, vehicles TOT 8 ACT 8 B
(pid=19117) Step #1.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 8 ACT 8 BUF 0)           
(pid=19117) Step #2.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 8 ACT 8 BUF 0)           
(pid=19117) Step #3.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19117) Step #4.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19117) Step #5.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 16 ACT 16 BUF 0)         
(pid=19117) Step #6.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 24 ACT 24 BUF 0)         
(pid=19117) Step #7.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 24 ACT 24 BUF 0)         
(pid=19117) Step #8.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 24 ACT 24 BUF 0)         
(pid=19117) Step #9.00 (1ms ~= 1000.00*RT, ~32000.00UPS, TraCI: 1ms, vehicles TOT 32 ACT 3
(pid=19117) Step #10.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 32 ACT 32 BUF 0)        
(pid=19117) Step #11.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 32 ACT 32 BUF 0)        
(pid=19117) Step #12.00 (2ms ~= 500.00*RT, ~20000.00UPS, TraCI: 0ms, vehicles TOT 40 ACT 4
(pid=19117) Step #13.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 40 ACT 40 BUF 0)        
(pid=19117) Step #14.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 40 ACT 40 BUF 0)        
(pid=19117) Step #15.00 (2ms ~= 500.00*RT, ~24000.00UPS, TraCI: 0ms, vehicles TOT 48 ACT 4
(pid=19117) Step #16.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 48 ACT 48 BUF 0)        
(pid=19117) Step #17.00 (1ms ~= 1000.00*RT, ~48000.00UPS, TraCI: 0ms, vehicles TOT 48 ACT 
(pid=19117) Step #18.00 (2ms ~= 500.00*RT, ~28000.00UPS, TraCI: 0ms, vehicles TOT 56 ACT 5
(pid=19117) Step #19.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 56 ACT 56 BUF 0)        
(pid=19117) Step #20.00 (1ms ~= 1000.00*RT, ~56000.00UPS, TraCI: 0ms, vehicles TOT 56 ACT 
(pid=19117) Step #21.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 64 ACT 64 BUF 0)        
(pid=19117) Step #22.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 64 ACT 64 BUF 0)        
(pid=19117) Step #23.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 64 ACT 64 BUF 0)        
(pid=19117) Step #24.00 (1ms ~= 1000.00*RT, ~72000.00UPS, TraCI: 0ms, vehicles TOT 72 ACT 
(pid=19117) Step #25.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 72 ACT 72 BUF 0)        
(pid=19117) Step #26.00 (1ms ~= 1000.00*RT, ~72000.00UPS, TraCI: 0ms, vehicles TOT 72 ACT 
(pid=19117) Step #27.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 80 ACT 80 BUF 0)        
(pid=19117) Step #28.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 80 ACT 80 BUF 0)        
(pid=19117) Step #29.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 80 ACT 80 BUF 0)        
(pid=19117) Step #30.00 (1ms ~= 1000.00*RT, ~88000.00UPS, TraCI: 0ms, vehicles TOT 88 ACT 
(pid=19117) Step #31.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 88 ACT 88 BUF 0)        
(pid=19117) Step #32.00 (1ms ~= 1000.00*RT, ~88000.00UPS, TraCI: 0ms, vehicles TOT 88 ACT 
(pid=19117) Step #33.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 96 ACT 96 BUF 0)        
(pid=19117) Step #34.00 (1ms ~= 1000.00*RT, ~96000.00UPS, TraCI: 0ms, vehicles TOT 96 ACT 
(pid=19117) Step #35.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 96 ACT 96 BUF 0)        
(pid=19117) Step #36.00 (1ms ~= 1000.00*RT, ~104000.00UPS, TraCI: 0ms, vehicles TOT 104 AC
(pid=19117) Step #37.00 (1ms ~= 1000.00*RT, ~104000.00UPS, TraCI: 0ms, vehicles TOT 104 AC
(pid=19117) Step #38.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 104 ACT 104 BUF 0)      
(pid=19117) Step #39.00 (1ms ~= 1000.00*RT, ~112000.00UPS, TraCI: 0ms, vehicles TOT 112 AC
(pid=19117) Step #40.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 112 ACT 112 BUF 0)      
(pid=19117) Step #41.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 112 ACT 112 BUF 0)      
(pid=19117) Step #42.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 120 ACT 120 BUF 0)      
(pid=19117) Step #43.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 120 ACT 120 BUF 0)      
(pid=19117) Step #44.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 120 ACT 120 BUF 0)      
(pid=19117) Step #45.00 (1ms ~= 1000.00*RT, ~128000.00UPS, TraCI: 0ms, vehicles TOT 128 AC
(pid=19117) Step #46.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 128 ACT 128 BUF 0)      
(pid=19117) Step #47.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 128 ACT 128 BUF 0)      
(pid=19117) Step #48.00 (1ms ~= 1000.00*RT, ~136000.00UPS, TraCI: 0ms, vehicles TOT 136 AC
(pid=19117) Step #49.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 136 ACT 136 BUF 0)      
(pid=19117) Step #50.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 136 ACT 136 BUF 0)      
(pid=19117) Step #51.00 (1ms ~= 1000.00*RT, ~144000.00UPS, TraCI: 0ms, vehicles
(pid=19115)  TOT 144 AC
(pid=19115) Step #52.00 (1ms ~= 1000.00*RT, ~144000.00UPS, TraCI: 0ms, vehicles TOT 144 AC
(pid=19115) Step #53.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 144 ACT 144 BUF 0)      
(pid=19115) Step #54.00 (1ms ~= 1000.00*RT, ~152000.00UPS, TraCI: 0ms, vehicles TOT 152 AC
(pid=19115) Step #55.00 (1ms ~= 1000.00*RT, ~152000.00UPS, TraCI: 0ms, vehicles TOT 152 AC
(pid=19115) Step #56.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 152 ACT 152 BUF 0)      
(pid=19115) Step #57.00 (1ms ~= 1000.00*RT, ~160000.00UPS, TraCI: 0ms, vehicles TOT 160 AC
(pid=19115) Step #58.00 (1ms ~= 1000.00*RT, ~160000.00UPS, TraCI: 0ms, vehicles TOT 160 AC
(pid=19115) Step #59.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 160 ACT 160 BUF 0)      
(pid=19115) Step #60.00 (1ms ~= 1000.00*RT, ~168000.00UPS, TraCI: 0ms, vehicles TOT 168 AC
(pid=19115) Step #61.00 (1ms ~= 1000.00*RT, ~168000.00UPS, TraCI: 0ms, vehicles TOT 168 AC
(pid=19115) Step #62.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 168 ACT 168 BUF 0)      
(pid=19115) Step #63.00 (1ms ~= 1000.00*RT, ~176000.00UPS, TraCI: 0ms, vehicles TOT 176 AC
(pid=19115) Step #64.00 (1ms ~= 1000.00*RT, ~176000.00UPS, TraCI: 0ms, vehicles TOT 176 AC
(pid=19115) Step #65.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 176 ACT 176 BUF 0)      
(pid=19115) Step #66.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 184 ACT 184 BUF 0)      
(pid=19115) Step #67.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 184 ACT 184 BUF 0)      
(pid=19115) Step #68.00 (1ms ~= 1000.00*RT, ~184000.00UPS, TraCI: 0ms, vehicles TOT 184 AC
(pid=19115) Step #69.00 (1ms ~= 1000.00*RT, ~192000.00UPS, TraCI: 0ms, vehicles TOT 192 AC
(pid=19115) Step #70.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 192 ACT 192 BUF 0)      
(pid=19115) Step #71.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 192 ACT 192 BUF 0)      
(pid=19115) Step #72.00 (1ms ~= 1000.00*RT, ~200000.00UPS, TraCI: 0ms, vehicles TOT 200 AC
(pid=19115) Step #73.00 (1ms ~= 1000.00*RT, ~200000.00UPS, TraCI: 0ms, vehicles TOT 200 AC
(pid=19115) Step #74.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 200 ACT 200 BUF 0)      
(pid=19115) Step #75.00 (1ms ~= 1000.00*RT, ~208000.00UPS, TraCI: 0ms, vehicles TOT 208 AC
(pid=19115) Step #76.00 (1ms ~= 1000.00*RT, ~208000.00UPS, TraCI: 0ms, vehicles TOT 208 AC
(pid=19115) Step #77.00 (1ms ~= 1000.00*RT, ~208000.00UPS, TraCI: 0ms, vehicles TOT 208 AC
(pid=19115) Step #78.00 (1ms ~= 1000.00*RT, ~216000.00UPS, TraCI: 0ms, vehicles TOT 216 AC
(pid=19115) Step #79.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 216 ACT 216 BUF 0)      
(pid=19115) Step #80.00 (1ms ~= 1000.00*RT, ~216000.00UPS, TraCI: 0ms, vehicles TOT 216 AC
(pid=19115) Step #81.00 (1ms ~= 1000.00*RT, ~224000.00UPS, TraCI: 0ms, vehicles TOT 224 AC
(pid=19115) Step #82.00 (1ms ~= 1000.00*RT, ~224000.00UPS, TraCI: 0ms, vehicles TOT 224 AC
(pid=19115) Step #83.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 224 ACT 224 BUF 0)      
(pid=19115) Step #84.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 232 ACT 232 BUF 0)      
(pid=19115) Step #85.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 232 ACT 232 BUF 0)      
(pid=19115) Step #86.00 (1ms ~= 1000.00*RT, ~232000.00UPS, TraCI: 0ms, vehicles TOT 232 AC
(pid=19115) Step #87.00 (1ms ~= 1000.00*RT, ~240000.00UPS, TraCI: 0ms, vehicles TOT 240 AC
(pid=19115) Step #88.00 (1ms ~= 1000.00*RT, ~240000.00UPS, TraCI: 0ms, vehicles TOT 240 AC
(pid=19115) Step #89.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 240 ACT 240 BUF 0)      
(pid=19115) Step #90.00 (1ms ~= 1000.00*RT, ~248000.00UPS, TraCI: 1ms, vehicles TOT 248 AC
(pid=19115) Step #91.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 248 ACT 248 BUF 0)      
(pid=19115) Step #92.00 (1ms ~= 1000.00*RT, ~248000.00UPS, TraCI: 0ms, vehicles TOT 248 AC
(pid=19115) Step #93.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19115) Step #94.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19115) Step #95.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19115) Step #96.00 (1ms ~= 1000.00*RT, ~264000.00UPS, TraCI: 0ms, vehicles TOT 264 AC
(pid=19115) Step #97.00 (1ms ~= 1000.00*RT, ~264000.00UPS, TraCI: 0ms, vehicles TOT 264 AC
(pid=19115) Step #98.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 264 ACT 264 BUF 0)      
(pid=19115) Step #99.00 (1ms ~= 1000.00*RT, ~272000.00UPS, TraCI: 0ms, vehicles TOT 272 AC
(pid=19115) Step #100.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 272 ACT 272 BUF 0)     
(pid=19115) Step #101.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 272 ACT 272 BUF 0)     
(pid=19115) Step #102.00 (1ms ~= 1000.00*RT, ~280000.00UPS, TraCI: 0ms, vehicles TOT 280 A
(pid=19115) Step #103.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 
(pid=19117) Exception in thread Thread-1:
(pid=19117) Traceback (most recent call last):
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 47, in get
(pid=19117)     self.feed_dict, os.environ.get("TF_TIMELINE_DIR"))
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 85, in run_timeline
(pid=19117)     fetches = sess.run(ops, feed_dict=feed_dict)
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
(pid=19117)     run_metadata_ptr)
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
(pid=19117)     str(subfeed_t.get_shape())))
(pid=19117) ValueError: Cannot feed value of shape (16, 11) for Tensor '0/Placeholder:0', which has shape '(?, 6)'
(pid=19117) 
(pid=19117) During handling of the above exception, another exception occurred:
(pid=19117) 
(pid=19117) Traceback (most recent call last):
(pid=19117)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=19117)     self.run()
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 148, in run
(pid=19117)     raise e
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 145, in run
(pid=19117)     self._run()
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 167, in _run
(pid=19117)     item = next(rollout_provider)
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 316, in _env_runner
(pid=19117)     active_episodes)
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 515, in _do_policy_eval
(pid=19117)     eval_results[k] = builder.get(v)
(pid=19117)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 50, in get
(pid=19117)     self.fetches, self.feed_dict))
(pid=19117) ValueError: Error fetching: [<tf.Tensor '0/Squeeze:0' shape=(?,) dtype=int64>, {'action_prob': <tf.Tensor '0/Exp_1:0' shape=(?,) dtype=float32>, 'vf_preds': <tf.Tensor '0/Reshape:0' shape=(?,) dtype=float32>}], feed_dict={<tf.Tensor '0/Placeholder:0' shape=(?, 6) dtype=float32>: [array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19117)        0.9510391 , 0.9510391 , 0.95927297, 0.95927297, 0.9510391 ,
(pid=19117)        0.9510391 ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19117)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19117)        0.        ]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19117)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19117)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19117)        0.        ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19117)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19117)        0.        ]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19117)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19117)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])], <tf.Tensor '0/action:0' shape=(?,) dtype=int64>: [array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0)], <tf.Tensor '0/prev_reward:0' shape=(?,) dtype=float32>: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], <tf.Tensor '0/PlaceholderWithDefault:0' shape=() dtype=bool>: False}
(pid=19117) 
(pid=19117)  TOT 144 AC
(pid=19117) Step #52.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 144 ACT 144 BUF 0)      
(pid=19117) Step #53.00 (1ms ~= 1000.00*RT, ~144000.00UPS, TraCI: 0ms, vehicles TOT 144 AC
(pid=19117) Step #54.00 (3ms ~= 333.33*RT, ~50666.67UPS, TraCI: 0ms, vehicles TOT 152 ACT 
(pid=19117) Step #55.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 152 ACT 152 BUF 0)      
(pid=19117) Step #56.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 152 ACT 152 BUF 0)      
(pid=19117) Step #57.00 (1ms ~= 1000.00*RT, ~160000.00UPS, TraCI: 0ms, vehicles TOT 160 AC
(pid=19117) Step #58.00 (1ms ~= 1000.00*RT, ~160000.00UPS, TraCI: 0ms, vehicles TOT 160 AC
(pid=19117) Step #59.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 160 ACT 160 BUF 0)      
(pid=19117) Step #60.00 (1ms ~= 1000.00*RT, ~168000.00UPS, TraCI: 0ms, vehicles TOT 168 AC
(pid=19117) Step #61.00 (1ms ~= 1000.00*RT, ~168000.00UPS, TraCI: 0ms, vehicles TOT 168 AC
(pid=19117) Step #62.00 (1ms ~= 1000.00*RT, ~168000.00UPS, TraCI: 0ms, vehicles TOT 168 AC
(pid=19117) Step #63.00 (1ms ~= 1000.00*RT, ~176000.00UPS, TraCI: 0ms, vehicles TOT 176 AC
(pid=19117) Step #64.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 176 ACT 176 BUF 0)      
(pid=19117) Step #65.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 176 ACT 176 BUF 0)      
(pid=19117) Step #66.00 (1ms ~= 1000.00*RT, ~184000.00UPS, TraCI: 0ms, vehicles TOT 184 AC
(pid=19117) Step #67.00 (1ms ~= 1000.00*RT, ~184000.00UPS, TraCI: 0ms, vehicles TOT 184 AC
(pid=19117) Step #68.00 (1ms ~= 1000.00*RT, ~184000.00UPS, TraCI: 0ms, vehicles TOT 184 AC
(pid=19117) Step #69.00 (1ms ~= 1000.00*RT, ~192000.00UPS, TraCI: 0ms, vehicles TOT 192 AC
(pid=19117) Step #70.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 192 ACT 192 BUF 0)      
(pid=19117) Step #71.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 192 ACT 192 BUF 0)      
(pid=19117) Step #72.00 (1ms ~= 1000.00*RT, ~200000.00UPS, TraCI: 0ms, vehicles TOT 200 AC
(pid=19117) Step #73.00 (1ms ~= 1000.00*RT, ~200000.00UPS, TraCI: 0ms, vehicles TOT 200 AC
(pid=19117) Step #74.00 (1ms ~= 1000.00*RT, ~200000.00UPS, TraCI: 0ms, vehicles TOT 200 AC
(pid=19117) Step #75.00 (1ms ~= 1000.00*RT, ~208000.00UPS, TraCI: 0ms, vehicles TOT 208 AC
(pid=19117) Step #76.00 (1ms ~= 1000.00*RT, ~208000.00UPS, TraCI: 0ms, vehicles TOT 208 AC
(pid=19117) Step #77.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 208 ACT 208 BUF 0)      
(pid=19117) Step #78.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 216 ACT 216 BUF 0)      
(pid=19117) Step #79.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 216 ACT 216 BUF 0)      
(pid=19117) Step #80.00 (1ms ~= 1000.00*RT, ~216000.00UPS, TraCI: 0ms, vehicles TOT 216 AC
(pid=19117) Step #81.00 (1ms ~= 1000.00*RT, ~224000.00UPS, TraCI: 0ms, vehicles TOT 224 AC
(pid=19117) Step #82.00 (1ms ~= 1000.00*RT, ~224000.00UPS, TraCI: 0ms, vehicles TOT 224 AC
(pid=19117) Step #83.00 (1ms ~= 1000.00*RT, ~224000.00UPS, TraCI: 0ms, vehicles TOT 224 AC
(pid=19117) Step #84.00 (1ms ~= 1000.00*RT, ~232000.00UPS, TraCI: 0ms, vehicles TOT 232 AC
(pid=19117) Step #85.00 (0ms ?*RT. ?UPS, TraCI: 0ms, vehicles TOT 232 ACT 232 BUF 0)      
(pid=19117) Step #86.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 232 ACT 232 BUF 0)      
(pid=19117) Step #87.00 (1ms ~= 1000.00*RT, ~240000.00UPS, TraCI: 0ms, vehicles TOT 240 AC
(pid=19117) Step #88.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 240 ACT 240 BUF 0)      
(pid=19117) Step #89.00 (1ms ~= 1000.00*RT, ~240000.00UPS, TraCI: 0ms, vehicles TOT 240 AC
(pid=19117) Step #90.00 (1ms ~= 1000.00*RT, ~248000.00UPS, TraCI: 0ms, vehicles TOT 248 AC
(pid=19117) Step #91.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 248 ACT 248 BUF 0)      
(pid=19117) Step #92.00 (1ms ~= 1000.00*RT, ~248000.00UPS, TraCI: 0ms, vehicles TOT 248 AC
(pid=19117) Step #93.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19117) Step #94.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19117) Step #95.00 (1ms ~= 1000.00*RT, ~256000.00UPS, TraCI: 0ms, vehicles TOT 256 AC
(pid=19117) Step #96.00 (1ms ~= 1000.00*RT, ~264000.00UPS, TraCI: 0ms, vehicles TOT 264 AC
(pid=19117) Step #97.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 264 ACT 264 BUF 0)      
(pid=19117) Step #98.00 (1ms ~= 1000.00*RT, ~264000.00UPS, TraCI: 0ms, vehicles TOT 264 AC
(pid=19117) Step #99.00 (1ms ~= 1000.00*RT, ~272000.00UPS, TraCI: 0ms, vehicles TOT 272 AC
(pid=19117) Step #100.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 272 ACT 272 BUF 0)     
(pid=19117) Step #101.00 (1ms ~= 1000.00*RT, ~272000.00UPS, TraCI: 0ms, vehicles TOT 272 A
(pid=19117) Step #102.00 (1ms ~= 1000.00*RT, ~280000.00UPS, TraCI: 0ms, vehicles TOT 280 A
(pid=19117) Step #103.00 (0ms ?*RT. ?UPS, TraCI: 1ms, vehicles TOT 
2019-03-29 03:35:21,046 INFO agent.py:313 -- Worker crashed during call to train(). To attempt to continue training without the failed worker, set `'ignore_worker_failures': True`.
(pid=19115) Exception in thread Thread-1:
(pid=19115) Traceback (most recent call last):
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 47, in get
(pid=19115)     self.feed_dict, os.environ.get("TF_TIMELINE_DIR"))
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 85, in run_timeline
(pid=19115)     fetches = sess.run(ops, feed_dict=feed_dict)
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
(pid=19115)     run_metadata_ptr)
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
(pid=19115)     str(subfeed_t.get_shape())))
(pid=19115) ValueError: Cannot feed value of shape (16, 11) for Tensor '0/Placeholder:0', which has shape '(?, 6)'
(pid=19115) 
(pid=19115) During handling of the above exception, another exception occurred:
(pid=19115) 
(pid=19115) Traceback (most recent call last):
(pid=19115)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=19115)     self.run()
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 148, in run
(pid=19115)     raise e
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 145, in run
(pid=19115)     self._run()
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 167, in _run
(pid=19115)     item = next(rollout_provider)
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 316, in _env_runner
(pid=19115)     active_episodes)
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 515, in _do_policy_eval
(pid=19115)     eval_results[k] = builder.get(v)
(pid=19115)   File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 50, in get
(pid=19115)     self.fetches, self.feed_dict))
(pid=19115) ValueError: Error fetching: [<tf.Tensor '0/Squeeze:0' shape=(?,) dtype=int64>, {'action_prob': <tf.Tensor '0/Exp_1:0' shape=(?,) dtype=float32>, 'vf_preds': <tf.Tensor '0/Reshape:0' shape=(?,) dtype=float32>}], feed_dict={<tf.Tensor '0/Placeholder:0' shape=(?, 6) dtype=float32>: [array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19115)        0.9510391 , 0.9510391 , 0.95927297, 0.95927297, 0.9510391 ,
(pid=19115)        0.9510391 ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19115)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19115)        0.        ]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19115)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19115)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19115)        0.        ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
(pid=19115)        0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
(pid=19115)        0.        ]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19115)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
(pid=19115)        0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])], <tf.Tensor '0/action:0' shape=(?,) dtype=int64>: [array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0)], <tf.Tensor '0/prev_reward:0' shape=(?,) dtype=float32>: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], <tf.Tensor '0/PlaceholderWithDefault:0' shape=() dtype=bool>: False}
(pid=19115) 
Traceback (most recent call last):
  File "experiments/a3c_4x4grid.py", line 49, in <module>
    print(trainer.train())  # distributed training step
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/agents/agent.py", line 316, in train
    raise e
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/agents/agent.py", line 305, in train
    result = Trainable.train(self)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/tune/trainable.py", line 151, in train
    result = self._train()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/agents/a3c/a3c.py", line 71, in _train
    self.optimizer.step()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/optimizers/async_gradients_optimizer.py", line 50, in step
    gradient, info = ray.get(future)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/worker.py", line 2316, in get
    raise value
ray.exceptions.RayTaskError: ray_PolicyEvaluator:compute_gradients() (pid=19117, host=dellg5)
ray.exceptions.RayTaskError: ray_PolicyEvaluator:sample() (pid=19117, host=dellg5)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 47, in get
    self.feed_dict, os.environ.get("TF_TIMELINE_DIR"))
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 85, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/fady/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (16, 11) for Tensor '0/Placeholder:0', which has shape '(?, 6)'

During handling of the above exception, another exception occurred:

ray_PolicyEvaluator:sample() (pid=19117, host=dellg5)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/policy_evaluator.py", line 393, in sample
    batches = [self.input_reader.next()]
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 41, in next
    batches = [self.get_data()]
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 180, in get_data
    raise rollout
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 148, in run
    raise e
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 145, in run
    self._run()
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 167, in _run
    item = next(rollout_provider)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 316, in _env_runner
    active_episodes)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 515, in _do_policy_eval
    eval_results[k] = builder.get(v)
  File "/home/fady/.local/lib/python3.6/site-packages/ray/rllib/utils/tf_run_builder.py", line 50, in get
    self.fetches, self.feed_dict))
ValueError: Error fetching: [<tf.Tensor '0/Squeeze:0' shape=(?,) dtype=int64>, {'action_prob': <tf.Tensor '0/Exp_1:0' shape=(?,) dtype=float32>, 'vf_preds': <tf.Tensor '0/Reshape:0' shape=(?,) dtype=float32>}], feed_dict={<tf.Tensor '0/Placeholder:0' shape=(?, 6) dtype=float32>: [array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.9510391 , 0.9510391 , 0.95927297, 0.95927297, 0.9510391 ,
       0.9510391 ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1.        , 0.        , 0.        , 0.95927297, 0.95927297,
       0.        , 0.        , 0.95927297, 0.95927297, 0.        ,
       0.        ]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), array([1.       , 0.       , 0.       , 0.       , 0.       , 0.9510391,
       0.9510391, 0.       , 0.       , 0.9510391, 0.9510391]), array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])], <tf.Tensor '0/action:0' shape=(?,) dtype=int64>: [array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0), array(0)], <tf.Tensor '0/prev_reward:0' shape=(?,) dtype=float32>: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], <tf.Tensor '0/PlaceholderWithDefault:0' shape=() dtype=bool>: False}


Step #120.00Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown TOT 296 A
Quitting (on error).

a single-agent

Thanks to the contribution of the author, this open source code is a great contribution to the community. Is there an example of running a single-agent using RLlib?

Error in plotting time delay vs iterations graph

I am able to run the ql_single-intersection.py completely without any error. While running the plot.py in the outputs folder the following error occurs. It shows a KeyError for the total_wating_time column generated as a result of running the ql_single-intersection.py

Mac Setup

Is there a way to run the package on MacOS? There seems to be some installation error.

Building wheels for collected packages: atari-py
  Building wheel for atari-py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/opt/[email protected]/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-install-ncx9h45m/atari-py/setup.py'"'"'; __file__='"'"'/private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-install-ncx9h45m/atari-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-wheel-h7mwui__
       cwd: /private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-install-ncx9h45m/atari-py/
  Complete output (502 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.15-x86_64-3.8
  creating build/lib.macosx-10.15-x86_64-3.8/atari_py
  copying atari_py/__init__.py -> build/lib.macosx-10.15-x86_64-3.8/atari_py
  copying atari_py/ale_python_interface.py -> build/lib.macosx-10.15-x86_64-3.8/atari_py
....
....
 copying atari_py/package_data.txt -> build/lib.macosx-10.15-x86_64-3.8/atari_py
  creating build/lib.macosx-10.15-x86_64-3.8/atari_py/tests
  copying atari_py/tests/__init__.py -> build/lib.macosx-10.15-x86_64-3.8/atari_py/tests
  copying atari_py/tests/test_smoke.py -> build/lib.macosx-10.15-x86_64-3.8/atari_py/tests
  running build_ext
  Traceback (most recent call last):
    File "/private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-install-ncx9h45m/atari-py/setup.py", line 22, in run
      subprocess.check_call(['cmake', '..'], cwd=cwd)
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 359, in check_call
      retcode = call(*popenargs, **kwargs)
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 340, in call
      with Popen(*popenargs, **kwargs) as p:
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'cmake'

...
File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/7r/4053h7154wv3qw4pnqw4w8l40000gq/T/pip-install-ncx9h45m/atari-py/setup.py", line 28, in run
      sys.stderr.write("Unable to execute '{}'. HINT: are you sure `make` is installed?\n".format(' '.join(cmd)))
NameError: name 'cmd' is not defined
  ----------------------------------------
  ERROR: Failed building wheel for atari-py
  Running setup.py clean for atari-py
Failed to build atari-py
Installing collected packages: atari-py, ray, sumo-rl
    Running setup.py install for atari-py ... error

Error of running a3c_4*4grid.py about ray

When I run a3c_4*4grid.py, it occurs that:

Exception has occurred: RayTaskError
�[36mray::RolloutWorker.foreach_policy()�[39m (pid=590, ip=192.168.1.106)
File "python/ray/_raylet.pyx", line 450, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 453, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 343, in ray._raylet.raise_if_dependency_failed
ray.exceptions.RaySystemError: System error: No module named 'environment'
traceback: Traceback (most recent call last):
File "/home/lyl/miniconda3/lib/python3.8/site-packages/ray/serialization.py", line 242, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
File "/home/lyl/miniconda3/lib/python3.8/site-packages/ray/serialization.py", line 188, in _deserialize_object
return self._deserialize_msgpack_data(data, metadata_fields)
File "/home/lyl/miniconda3/lib/python3.8/site-packages/ray/serialization.py", line 166, in _deserialize_msgpack_data
python_objects = self._deserialize_pickle5_data(pickle5_data)
File "/home/lyl/miniconda3/lib/python3.8/site-packages/ray/serialization.py", line 156, in _deserialize_pickle5_data
obj = pickle.loads(in_band)
ModuleNotFoundError: No module named 'environment'
File "/mnt/d/try/sumo-rl-master/experiments/a3c_4x4grid.py", line 32, in
trainer = A3CTrainer(env="4x4grid", config={

Can you help me how to solve it? Thanks.

Problem with sumo-gui

Hello I'm interested in your study and researching related studies.
So when I tried to run with 'use_gui = True ' , then the error message came out.
'UserWarning: Could not connect to TraCI sever using port xxxxx...'
I'm not sure where is the problem maybe I have not enough information about SUMO.

Can you help me with this problem?

Documentation for Different Environments

Hey,

I was planning to explore using a handful of these environments as a part of my research. However, unless I'm missing something, there's no explanation or visuals of the mechanics or behaviors of the different environments/maps? Is that the case, and if so would you be willing to take an hour to add it to the readme or something? It'd be super helpful for those potentially interested in your environment.

TypeError: 'int' object is not iterable. Maybe caused by an imcompatible version of sumo

Hi, when I used sumo-1.1.0 to run this ql_single-intersection.py, it didn't work well. The version of my computer system is ubuntu 16.04, and the version of python is 3.5 which was created by Anaconda.

The traceback was reported by traci. So I wondered this error was caused by an imcompatible version of sumo.

Which version of sumo do you use?

I followed the tutorial in the readme and tried

python ql_single-intersection.py

but it told me -route is needed. So I tried the following command.

$ ~/Downloads/Compressed/sumo-rl-master$ python experiments/ql_single-intersection.py  -route nets/single-intersection/single-intersection.rou.xml

The error messages followed.

WARNING: Not monitoring node memory since psutil is not installed. Install this with pip install psutil (or ray[debug]) to enable debugging of memory-related crashes.
Retrying in 1 seconds
/home/user/anaconda3/envs/py3_5/lib/python3.5/site-packages/gym/logger.py:30: UserWarning: WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Retrying in 1 seconds?UPS, TraCI: 3ms, vehicles TOT 0 ACT 0 BUF 0)
Traceback (most recent call last):
File "ql_single-intersection.py", line 68, in
initial_states = env.reset()
File "/home/user/Downloads/Compressed/sumo-rl-master/sumo_rl/environment/env.py", line 105, in reset
self.traffic_signals[ts] = TrafficSignal(self, ts, self.delta_time, self.min_green, self.max_green, self.phases)
File "/home/user/Downloads/Compressed/sumo-rl-master/sumo_rl/environment/traffic_signal.py", line 31, in init
traci.trafficlight.setCompleteRedYellowGreenDefinition(self.id, logic)
File "/usr/share/sumo/tools/traci/_trafficlight.py", line 256, in setCompleteRedYellowGreenDefinition
for p in tls.phases:
TypeError: 'int' object is not iterable
Step #0.00Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown
Quitting (on error).

problem while running a3c_4x4grid

Hey,

When I run a3c_4x4grid.py, I get the error

sjj@sjj-virtual-machine:~/sumo-rl$ python3 experiments/a3c_4x4grid.py
2021-10-25 14:00:47,456 INFO services.py:1092 -- View the Ray dashboard at http://127.0.0.1:8265
/home/sjj/.local/lib/python3.6/site-packages/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
"Box bound precision lowered by casting to {}".format(self.dtype)
2021-10-25 14:00:50,149 INFO trainer.py:1065 -- _use_trajectory_view_api only supported for PyTorch so far! Will run w/o.
2021-10-25 14:00:50,149 INFO trainer.py:619 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(pid=2043) Retrying in 1 seconds
(pid=2044) Retrying in 1 seconds
(pid=2043) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 55ms, vehicles TOT 0 ACT 0 BUF 0)
(pid=2043) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/wrappers/base.py:59: UserWarning: The action_spaces dictionary is deprecated. Use the action_space function instead.
(pid=2043) warnings.warn("The action_spaces dictionary is deprecated. Use the action_space function instead.")
(pid=2043) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/env.py:89: UserWarning: Your environment should override the action_space function. Attempting to use the action_spaces dict attribute.
(pid=2043) warnings.warn("Your environment should override the action_space function. Attempting to use the action_spaces dict attribute.")
(pid=2043) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/wrappers/base.py:51: UserWarning: The observation_spaces dictionary is deprecated. Use the observation_space function instead.
(pid=2043) warnings.warn("The observation_spaces dictionary is deprecated. Use the observation_space function instead.")
(pid=2043) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/env.py:78: UserWarning: Your environment should override the observation_space function. Attempting to use the observation_spaces dict attribute.
(pid=2043) warnings.warn("Your environment should override the observation_space function. Attempting to use the observation_spaces dict attribute.")
(pid=2044) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/wrappers/base.py:59: UserWarning: The action_spaces dictionary is deprecated. Use the action_space function instead.
(pid=2044) warnings.warn("The action_spaces dictionary is deprecated. Use the action_space function instead.")
(pid=2044) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/env.py:89: UserWarning: Your environment should override the action_space function. Attempting to use the action_spaces dict attribute.
(pid=2044) warnings.warn("Your environment should override the action_space function. Attempting to use the action_spaces dict attribute.")
(pid=2044) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/wrappers/base.py:51: UserWarning: The observation_spaces dictionary is deprecated. Use the observation_space function instead.
(pid=2044) warnings.warn("The observation_spaces dictionary is deprecated. Use the observation_space function instead.")
(pid=2044) /home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/env.py:78: UserWarning: Your environment should override the observation_space function. Attempting to use the observation_spaces dict attribute.
(pid=2044) warnings.warn("Your environment should override the observation_space function. Attempting to use the observation_spaces dict attribute.")
(pid=2044) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 57ms, vehicles TOT 0 ACT 0 BUF 0)
Traceback (most recent call last):
File "experiments/a3c_4x4grid.py", line 38, in
"no_done_at_end": True
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 106, in init
Trainer.init(self, config, env, logger_creator)
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 477, in init
super().init(config, logger_creator)
File "/home/sjj/.local/lib/python3.6/site-packages/ray/tune/trainable.py", line 249, in init
self.setup(copy.deepcopy(self.config))
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 630, in setup
self._init(self.config, self.env_creator)
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 138, in _init
num_workers=self.config["num_workers"])
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 707, in _make_workers
logdir=self.logdir)
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 81, in init
lambda p, pid: (pid, p.observation_space, p.action_space)))
File "/home/sjj/.local/lib/python3.6/site-packages/ray/worker.py", line 1452, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray::RolloutWorker.foreach_policy() (pid=2044, ip=192.168.52.129)
File "python/ray/_raylet.pyx", line 443, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 477, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 481, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 482, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 436, in ray._raylet.execute_task.function_executor
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 366, in init
self.env = _validate_env(env_creator(env_context))
File "experiments/a3c_4x4grid.py", line 28, in
max_depart_delay=0)))
File "/home/sjj/.local/lib/python3.6/site-packages/ray/rllib/env/pettingzoo_env.py", line 72, in init
self.agents = self.aec_env.agents
File "/home/sjj/.local/lib/python3.6/site-packages/pettingzoo/utils/wrappers/order_enforcing.py", line 38, in getattr
raise AttributeError(f"{value} cannot be accessed before reset")
AttributeError: agents cannot be accessed before reset

Can you please tell me how to make it work?

Error after 6 runs while running a2c_2way-single-intersection.py and dqn_2way-single-intersection.py

While running dqn_2way-single-intersection.py and a2c_2wy-single-intersection.py code after 6 runs I am getting the following error.
Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown 46390 ACT 56 BUF 2)
Quitting (on error).
How to fix this error?

lucasalegre / sumo-rl Goto Github PK

sumo-rl's Introduction

SUMO-RL

Install

Install SUMO latest version:

Install SUMO-RL

MDP - Observations, Actions and Rewards

Observation

Action

Rewards

API's (Gymnasium and PettingZoo)

Gymnasium Single-Agent API

PettingZoo Multi-Agent API

RESCO Benchmarks

Experiments

Q-learning in a one-way single intersection:

RLlib PPO multiagent in a 4x4 grid:

stable-baselines3 DQN in a 2-way single intersection:

Plotting results:

Citing

sumo-rl's People

Stargazers

Watchers

Forkers

sumo-rl's Issues

Recommend Projects

Recommend Topics

Recommend Org