By Yong Li, Jiabei Zeng, Shiguang Shan, Xilin Chen

News! 2019/02/25. We add the model config of paper: "Occlusion aware facial expression recognition using CNN with attention mechanism". All the details can be found in prototxt/gACNN_train.prototxt.

Code for our ICPR 2018 paper: "Patch-Gated CNN for Occlusion-aware Facial Expression Recognition". We designed a Patch-Gated CNN that can percept and ignore the occlusions for facial expression recognition. All the details can be found in prototxt/pACNN_train.prototxt.
Note that the code is based on caffe, a famous deep learning framework.

The order of the 68 facial landmarks can be found at Link.
The following figure shows how we select the ciritcal points to crop the patches from a facial image:
(a) denotes totally 30 landmarks within original 68 facial landmarks. These points are involved in point selection
(b) shows 16 points that we pick to cover the facial regions on or around eyes, eyebrows, nose, mouth
(c) illustrates four points that are recomputed to better cover eyes and eyebrows
(d) displays four points that are re-computed to cover facial cheeks
(e) shows the selected 24 facial landmarks, around which the patches in (f) are cropped

Training yourself:
  We designed a caffe layer named as multi_roi_pooling_layer. Currently the layer is provided with a GPU version.

  Building the multi_roi_pooling_layer with the related caffe.proto in proto folder, you can start training a model
   immediately.

Precautions:
  The training and testing image list should be arranged as:
    image_path   expression_label   point1_h   point1_w   pint2_h   point2_w   ...   point_24_h   point24_w

  An example of train.list has been provided in train_list_example folder.
  We provided a python script to convert 68 facial landmarks to desired 24 points, the file locate in convert_point folder.


Dateset resource:
  We collected and labelled a facial expression dataset (FED-RO) in the presence of real occlusions: . download link
  Alternative download link based on Baidu Yun. Extraction code: o5di

@inproceedings{li2018patch,
  title={Patch-Gated CNN for Occlusion-aware Facial Expression Recognition},
  author={Li, Yong and Zeng, Jiabei and Shan, Shiguang and Chen, Xilin},
  booktitle={ICPR},
  year={2018}
}

@article{li2019occlusion,
  title={Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism},
  author={Li, Yong and Zeng, Jiabei and Shan, Shiguang and Chen, Xilin},
  journal={IEEE Transactions on Image Processing},
  volume={28},
  number={5},
  pages={2439--2450},
  year={2019}
}