This project is the first project completed for Machine Learning and Data Mining course during my senior year at the University of Toronto. In this project, I explored the effectiveness of applying linear regression on classfications problems of two different actors, two different genders and the overfitting problems. I also visualized the final weights used on the feature vectors.

I would like to thank my collaborator, Yuan Yao, for his strong dedication and intelligence to make this project a great one.

The specific instructions of this project could be found in the instructions link. In this project, we used the classic MINST handwritten digits dataset and the facescrub dataset to explore the following aspects of deep neural networks

1. classify which class a handwritten digit belongs to using a single layer neural network, compare the results with the linear classifiers
2. classify which actor is in the image given a image data using two layer neural network with 6 classes
3. visualize the weights generated by the neural network of the features as images
4. build on the activation value of AlexNet, build a fully-connected neural network to classify the actors image

The !FaceScrub Dataset is downloaded from UC Irvine machine learning repository. The original images looks like the following after preprocessing.

The MINST dataset is directly read from the provided .mat file in this repository. The data looks like the following after preprocessiing.

The specific report of this project could be found in the deepnn.py file in this repository.

For the MINST dataset, the superior performance of single layer neural network to linear classifier is evident from the better performance in both the training set and the test set.

The performance on training set using logistic method is : 86.25%
The performance on test set using logistic method is : 83.3%

The performance on training set using linear method is : 73.2833%
The performance on test set using linear method is : 71.7%

For the FaceScrub Dataset, we built a two layer neural network with the following structure using Tensorflow

This neural network could perform relatively well on classifying on a more difficult problem than binary classification, yielding the following results

The accuracy on the training set: 99.85%
The accuracy on the test set : 87.22%
The accuracy on the validation set: 94.99%

The learning curve is shown as the following

To see the effect of overfitting, we randomly selecting a small set of images (120 images, with no balance of class) and set the image size as 64 X 64. With 4096 inputs and a small training set size, the model without regularizations will easily overt and produce a degraded performance

We set the learning rate as 0.0001, the results are shown below

The performance on test set without regularization is 73.3%
The performance on test set with regularization is 80.0%

The learning curve is shown in the following

AlexNet significantly improves the accuracy of the 6-actor classifcation problem, boosting the accuracy up to 96.11% on the test set. By using mini-batch training, the The learning curve with mini-batch training is shown below