By Jufeng Yang, Dongyu She, Ming Sun

We develop a multi-task deep framework by jointly optimizing classification and distribution prediction

• It achieves state-of-the-art performance on emotion classification, and LDL prediction tasks.
• Our code is based on Caffe.

The paper has been accepted by IJCAI 2017. For more details, please refer to our paper.

Our framework is released under the MIT License (refer to the LICENSE file for details).

If you find our framework useful in your research, please consider citing:

@inproceedings{ijcai2018joint,
	Author = {Jufeng Yang, Dongyu She, Ming Sun},
	Title = {Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network},
	booktitle = {IJCAI},
	Year = {2017}
}

1. Requirements: software
2. Requirements: hardware
3. Basic installation
4. Extra Downloads (dataset lmdb)
5. Usage
6. Trained models

1. Requirements for Caffe (see: Caffe installation instructions)

# This is required only if you will compile the matlab interface.

1. NVIDIA GTX TITANX (~12G of memory)

1. Clone the repository

git clone https://github.com/sherleens/EmotionDistributionLearning.git

2. Change the label number in $ROOT/caffe_KL/src/caffe/layers/data_layer.cpp

line52:    int labelNum = 7; //change to the corresponding label number with the LMDB file
...
line110:   int labelNum = 7; //for example, change to 7 for the Emotion6 dataset, 8 for the Flickr_LDL dataset 

3. Build Caffe with KLloss

cd $ROOT/caffe_KL
# Now follow the Caffe installation instructions here
#   http://caffe.berkeleyvision.org/installation.html
# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make all -j 8 && make matcaffe

LMDB file is generated by modified code (caffe_KL/tool/convert_imageset_r) to support multiple ground-truth labels, which can also be downloaded from Baidu, password:b83b.

Contents:

• dataset_config

  • emotion6_config.mat
  • flickrldl_config.mat
  • twitterldl_config.mat
  • readme

• lmdb_file

  • train_twitterldl_split1_lmdb.tar 1.5G
  • train_flickrldl_split1_lmdb.tar 1.67G
  • train_emotion6_lmdb.tar 265.4M
  • test_twitterldl_split1_lmdb.tar 384.7M
  • test_flickrldl_split1_lmdb.tar 426.8M
  • test_emotion6_lmdb.tar 113.7M

Train a deep network. For example, train a VGG16 network on distribution datasets.

cd prototxt && bash ./train_vgg_gs.sh

The models trained on the distribution datasets can be downloaded from here.

coming soon...