Implementation of visualization techniques for CNN in Caffe (t-SNE, DeconvNet, Image occlusions)

• Caffe and pyCaffe, Installation guide.

• numpy, scikit-image, sklearn, skdata

The file tsne.py contains an implementation of t-Stochastic Neighbor Embedding as described in the following paper: Visualizing Data using t-SNE.

It's possible to test this implementation stand-alone on the well known mnist dataset. It's necessary to download data and labels from here, put them in the same folder of the source code and run

    $ python tsne.py

To test how the implementation works with features extracted from a CNN, we will use the file tsneCNN.py.

The simplest way to test the code consists in downloading the standard CaffeNet network using

    $ wget -O path/to/caffe/models/bvls_reference_caffenet/bvlc_reference_caffenet.caffemodel http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel

Get the mapping from network output to synsets using

    $ ./path/to/caffe/data/ilsvrc12/data/get_ilsvrc_aux.sh

Download some synsets of images from ImageNet and place them in a folder called "image_dir". The script will take up to 150 images from each subfolder of "image_dir". It's assumed that each of these subfolder is a synset directory as downloaded from ImageNet.

Launch the script exploiting default parameters.

    $ python tsneCNN.py -i image_dir -g

It's possible to test different networks simply by downloading their prototxt and weights and providing the paths to these file when launching the script. Do not forget to set also the network type! This argument is used to choose from which layer features have to be extracted.

The expected output consists in:

• The 2-dimensional embedding of the extracted features (each image is represented as a "dot" coloured accoring to its real or predicted class). This allows to see how this unsupervised method performs really well regardless of the CNN prediction.

• A video showing how the embedding changes through the t-SNE iterations.

• The real images organized according to the embedding. This allows to see how t-SNE preserves local differences within the same class of images (N.B. Zoom on an area of the image to see something meaningful)

The file cnn_occlusion.py is an implementation of the occlusion technics described in the section 4.2 of the following paper: Visualizing and Understanding Convolutional Networks.

In the following, the procedure to use caffenet will be described. It is possible to use different models by changing the default parameters. The caffenet prototxt is already included in the caffe installation.

To download the caffemodel file in the proper place run:

    $ wget -O path/to/caffe/models/bvls_reference_caffenet/bvlc_reference_caffenet.caffemodel http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel

Choose a class from the file synset_words.txt and download at least one image from Image-Net. Suppose the name of the image is n01440764_18.JPEG. It's important to don't change the name of the image since it contains the synset reference.

To test the code:

    $ python cnn_occlusion.py /path/to/n01440764_18.JPEG  

You can expect something like this: