Language Conditioned Traffic Generation
Shuhan Tan¹, Boris Ivanovic², Xinshuo Weng², Marco Pavone², Philipp Krähenbühl¹

¹UT Austin, ² NVIDIA

Conference on Robot Learning (CoRL), 2023

Webpage | Video | Paper (Arxiv) | Demo (Colab)

• 2 Oct, 2023: Initial code release.
• 30 Aug, 2023: Our paper was accepted at CoRL 2023!
• 16 Jul, 2023: We released our paper on arXiv.

We provide an online demo in Colab. You can try it without any local installation. This demo includes:

1. Generate traffic scene with existing Structured Representation.
2. Generate traffic scene from existing LLM output.
3. Generate traffic scene with natural language and GPT-4 (requires OpenAI API Key).

Please also refer to the code inside for demonstrations of basic model usage.

# Clone the code to local
git clone https://github.com/Ariostgx/lctgen.git
cd lctgen

# Create virtual environment
conda create -n lctgen python=3.8
conda activate lctgen

# You should install pytorch by yourself to make them compatible with your GPU
pip install torch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 -f https://download.pytorch.org/whl/torch_stable.html

# Install other dependency
pip install -r requirements.txt

We provide a demo dataset and a pretrained model for quick start.

This is meant to be a quick demonstration of the model usage. For complete training and evaluation, please refer to the next section.

Please download the demo dataset of 30 scene clips demo_data.zip from here.

And then unzip all the files inside into data/demo/waymo folder.

Please download the example model checkpoint example.ckpt from here.

And then put it into checkpoints folder.

By default we use wandb to log the training process. You will need to login to wandb first.

You can also use tensorboard by setting _C.LOGGER to tsboard in lctgen/config/path_cfg.py.

python lctgen/main.py  --run-type eval --exp-config cfgs/demo_inference.yaml

python lctgen/main.py  --run-type train --exp-config cfgs/demo_train.yaml

We follow the data processing process in TrafficGen:

1. Download from Waymo Open Dataset:

• Register your Google account in: https://waymo.com/open/
• Open the following link with your Google account logged in: https://console.cloud.google.com/storage/browser/waymo_open_dataset_motion_v_1_1_0
• Download all the proto files from waymo_open_dataset_motion_v_1_1_0/uncompressed/scenario/training_20s
• Move download files to PATH_A, where you store the raw tf_record files.

2. Data Preprocess

python scripts/process_all_data.py PATH_A PATH_B

• Note: PATH_B is where you store the processed data.

3. Change _C.DATASET.DATA_PATH to PATH_B in lctgen/config/path_cfg.py.

python lctgen/main.py  --run-type eval --exp-config cfgs/inference.yaml

python lctgen/main.py  --run-type train --exp-config cfgs/train.yaml

We use code in TrafficGen for data processing and visualization. TrafficGen related code is in trafficgen folder.

• Add instructive scene editing.
• Add dataset of input text descriptions and LLM outputs.
• Initial repo & demo

We thank Yuxiao Chen, Yulong Cao, and Danfei Xu for their insightful discussions. This project is supported by the National Science Foundation under Grant No. IIS-1845485.

We also thank authors of TrafficGen for their open source code.

@article{tan2023lctgen,
  title={Language Conditioned Traffic Generation},
  author={Tan, Shuhan and Ivanovic, Boris and Weng, Xinshuo and Pavone, Marco and Kr\"ahenb\"uhl, Philipp,
  journal={7th Annual Conference on Robot Learning (CoRL)},
  year={2023}
}