This demo is based on Behaviour Cloning project from Term 1 of Udacity Self-Driving Car Engineer Nanodegree

The presentation slides are here

In this project we demonstrate few aspects of developing self-driving cars, which come together nicely in one simple project and are implemented using open source tools.

Setting: we have a car driving simulator. The simulator sends telemetery events with a picture from 'camera', the current steering angle and velocity. We write a program that processes the telemetery events, decides what steering angle and throttle to apply, and sends them to the simulator. The goal is to keep the car in the center of the road while driving around the track.

We can drive the car manually around the track and save the images and associated steering angles. This gives us labelled dataset which we use to train our program how to drive like a human. This is a supervised learning setting. We build a simple deep convolutional neural network (CNN) and train it to associate steering angle with an image.

Effectively the program clones driving behaviour. This approach to programming self-driving cars is called end-to-end learning because it takes raw sensor inputs (in this case image pixels) and produces the resulting control commands without breaking the intermediate steps down into logical steps like detecting lanes, objects, distances, applying if-then-else logic etc. This article by NVIDIA details how it was done successfully on a real car using NVIDIA Drive PX and about 72 hours of data, driving on different roads and in various weather conditions. They called their system DAVE 2 and you can see video of its performance here. Notice around minute 8 what image they send to the CNN and the feature maps it produces.

Here we repeat NVIDIA success to show that anyone can program self-driving cars using open source tools! We use a CNN architecture similar to NVIDIA's, code it up in Keras and tensorflow and then use python script as a glue to handle telemetery events using asincio, run the prediction and send the controls back.

The steps of this project are the following:

• Use the simulator to collect data of good driving behavior
• Design, train and validate a model that predicts a steering angle from image data
• Use the model to drive the vehicle autonomously around the first track in the simulator

Udacity has produced an open-source driving simulator based on Unity 3D engine. You can download it here (get Version 2 or above). You can even modify it for your needs if you feel adventurous!

After you start the simulator choose the desired graphics quality and resolution. Because later we will resize the images to 80x160 pixels, do not choose too high a resolution, as the saved images will take up more space and lenghten the preprocessing but will not improve the quiality of the results.

Press the record button and choose the folder where the simulator creates IMG folder and driving-log.csv files. Then steer the car around the track for data collection. Here are a few points to consider:

• steering with the keyboard (left/right buttons) is quite sensitive, so you may want to limit key press durations
• you can control steering via mouse instead of keyboard. This creates better angles for training. The angle is based on the mouse distance. To steer hold the left mouse button and move left or right. To reset the angle to 0 simply lift your finger off the left mouse button.
• the best data can be collected by using a joystick or a steering column, if you have them
• You can toggle record by pressing R key (or press record button on the screen)
• When recording is finished, all the captured images are saved to the disk at the same time You will see save status and play back of the captured data.
• You can takeover in autonomous mode. While W or S are held down you can control the car the same way you would in training mode. This can be helpful for debugging. As soon as W or S are let go simulator goes back into autonomous mode.
• Pressing the spacebar in training mode toggles cruise control on and off (effectively presses W for you).

Here is the video of how to record the data:

We use python 3.5 and conda environment/package manager

The steps are:

1. Install miniconda or full Anaconda on your computer
2. Setup conda environment
3. Each time you wish to work, activate your conda environment
4. Install tensorflow

Download the version of miniconda that matches your system. Make sure you download the version for Python 3.5.

Install miniconda on your machine. Detailed instructions:

• Linux: http://conda.pydata.org/docs/install/quick.html#linux-miniconda-install
• Mac: http://conda.pydata.org/docs/install/quick.html#os-x-miniconda-install
• Windows: http://conda.pydata.org/docs/install/quick.html#windows-miniconda-install

Create your the pydata-sdc conda environment.

conda env create -f environment.yml python=3.5

Verify that the carnd-term1 environment was created in your environments:

conda info --envs

Cleanup downloaded libraries (remove tarballs, zip files, etc):

conda clean -tp

To delete the environment when you dont need it:

conda env remove -n pydata-sdc

Now that you have created an environment, in order to use it, you will need to activate the environment. This must be done each time you begin a new working session i.e. open a new terminal window.

Activate the pydata-sdc environment:

On OS X and Linux:

$ source activate pydata-sdc

On Windows, depending on shell either:

$ source activate pydata-sdc

or

$ activate pydata-sdc

When you are in pydata-sdc virtual environment you see (pydata-sdc) at the beginning of the shell prompt.

If you DO have an NVIDIA GPU and CUDA drivers, install GPU version of tensorflow:

(pydata-sdc) $ pip install tensorflow-gpu==1.0.0

otherwise setup CPU version which should work on wide range of hardware:

(pydata-sdc) $ pip install tensorflow==1.0.0

We define and train our model in Jupyter notebook called behavioural-cloning.ipynb Start it up using the following command and then follow the instructions inside:

(pydata-sdc) $ jupyter notebook behavioural-cloning.ipynb

The runtime of this notebook for about 8000 images in IMG folder should be between 8 and 30 minutes, depending on hardware you use. If you use GPU the default batch size of 256 images requires that you have around 1gGb of GPU memory available for this notebook to run.

This is the original architecture of NVIDIA CNN:

We use a slightly modified version (sorry, did not have time to produce proper diagram)

Here is the video of how we train the model:

Now that we have a trained model (which is saved in model.h5 file) we can use it to drive the car! We use drive.py script that does it. The script does a few things:

1. load model from the file
2. starts telemetery server
3. initializes PI controller to adjust the throttle
4. processes incoming events and runs steering angle prediction

Here is the video of the car being driven by the deep net:

It looks a little wobbly, but that is just based on simple augmentation of driving data consisting of about 3 laps of driving. And using a simple deep neural net that was trained for just 5 epochs in 10 minutes. You can see the potential!

Note: There is known local system's setting issue with replacing "," with "." when using drive.py. When this happens it can make predicted steering values clipped to max/min values. If this occurs, a known fix for this is to add "export LANG=en_US.utf8" to the bashrc file.