This is a project to create a machine learning method for classification of the type of street sign that is shown in a image. The method will use transfer learning from a pre-trained convolutional neural network and training a classification model on the features obtained.

In general, training a good neural network to do image classification can take a lot of time and resources. Luckily there exists extremely complex networks that have been pre-trained to classify many different items. But what happens when we want to use this method on something the neural network has never seen before? It wouldn't be able to make these classifications.

This is where transfer learning comes in. Transfer learning is an extremely powerful tool in machine learning when it comes to utilizing neural networks. By passing an image through the network without retraining it, but stopping it before making its classification prediction, you can extract features that the network thought were important enough about your image. You can then train a classification model on these features with good results.

There is always need for new methods in recognition and classification of symbols and street signs for a number of uses including: self driving cars, street mapping, and driver safety systems. Classification has been done before, but there are no references that can be found for utilizing a transfer learning from a pre trained neural network.

The data used in this demonstration is the LISA traffic sign dataset that can be found here.

• As the image set is too large for this repo, the images are stored and accessed through AWS S3 bucket. The dataset comes with annotations for the location of signs within each image. A script, here, was created pull the images from the LISA file system and upload them to S3 Bucket as well as save a JSON file with file name, original path, bounding boxes, and sign label.

• A script was created here to pull each image from the bucket as well as the image dictionary created in the previous step, crop the images, pad them with black to square and then put them back on S3. The script has a lower threshold for images (12x12 pixels).

In all there were 15 classes with about 5500 images ranging from 12x12 to ~150x150 pixels. The classes are fairly unbalanced with stop signs making up the majority of the dataset.

• Using the cropped images dictionary created in the above steps, a train/test split is made using this script.

• The cropped images were passed through Inception V3 (here) and the bottleneck feature was extracted, which is a 2048 dimensional vector per image, and saved on S3.

• A multinomial logistic regression classifier was fit to the extracted features and saved as a pickle.

A web application was deployed utilizing the pickled model as well as extracted test features to make predictions on random images from the test set at: signedge.tech

• TSNE dimensionality reduction was done on the feature space to look at the groupings of the classes.

• A model that makes predictions off a small data set after using transfer learning.
• Building a website to house and show off the project.

Using the LIME method for explanation of where the predictive power comes from in my image set.

LISA Dataset:

Andreas Møgelmose, Mohan M. Trivedi, and Thomas B. Moeslund, "Vision based Traffic Sign Detection and Analysis for Intelligent Driver Assistance Systems: Perspectives and Survey," IEEE Transactions on Intelligent Transportation Systems, 2012.