Code Monkey home page Code Monkey logo

pg-cnn's Introduction

PG-CNN and its TIP version

By Yong Li, Jiabei Zeng, Shiguang Shan, Xilin Chen

News! 2019/02/25. We add the model config of paper: "Occlusion aware facial expression recognition using CNN with attention mechanism". All the details can be found in prototxt/gACNN_train.prototxt.


Code for our ICPR 2018 paper: "Patch-Gated CNN for Occlusion-aware Facial Expression Recognition". We designed a Patch-Gated CNN that can percept and ignore the occlusions for facial expression recognition. All the details can be found in prototxt/pACNN_train.prototxt.
Note that the code is based on caffe, a famous deep learning framework.

The order of the 68 facial landmarks can be found at Link.
The following figure shows how we select the ciritcal points to crop the patches from a facial image: alt text
(a) denotes totally 30 landmarks within original 68 facial landmarks. These points are involved in point selection
(b) shows 16 points that we pick to cover the facial regions on or around eyes, eyebrows, nose, mouth
(c) illustrates four points that are recomputed to better cover eyes and eyebrows
(d) displays four points that are re-computed to cover facial cheeks
(e) shows the selected 24 facial landmarks, around which the patches in (f) are cropped

Training yourself:
  We designed a caffe layer named as multi_roi_pooling_layer. Currently the layer is provided with a GPU version.

  Building the multi_roi_pooling_layer with the related caffe.proto in proto folder, you can start training a model
   immediately.


Precautions:
  The training and testing image list should be arranged as:
    image_path   expression_label   point1_h   point1_w   pint2_h   point2_w   ...   point_24_h   point24_w

  An example of train.list has been provided in train_list_example folder.
  We provided a python script to convert 68 facial landmarks to desired 24 points, the file locate in convert_point folder.


Dateset resource:
  We collected and labelled a facial expression dataset (FED-RO) in the presence of real occlusions: . download link
  Alternative download link based on Baidu Yun. Extraction code: o5di

Citation: If you use this code in your paper, please cite the following:

@inproceedings{li2018patch,
  title={Patch-Gated CNN for Occlusion-aware Facial Expression Recognition},
  author={Li, Yong and Zeng, Jiabei and Shan, Shiguang and Chen, Xilin},
  booktitle={ICPR},
  year={2018}
}
@article{li2019occlusion,
  title={Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism},
  author={Li, Yong and Zeng, Jiabei and Shan, Shiguang and Chen, Xilin},
  journal={IEEE Transactions on Image Processing},
  volume={28},
  number={5},
  pages={2439--2450},
  year={2019}
}

pg-cnn's People

Contributors

mysee1989 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

pg-cnn's Issues

Synthesis dataset

Your work is very good.But can you share the artificial synthesised datasets, such as occluded RAF-DB and affectnet?

Missing images in FEDRO dataset

Dear Sir,
I am testing your algorithm on FEDRO.
Only 381 images out of 400 are available in the dateset at the link provided. Missing images list is :
disgust/occ_img_45.jpg
disgust/occ_img_31.jpg
disgust/occ_img_33.jpg
disgust/occ_img_23.jpg
disgust/occ_img_47.jpg
fear/occ_img_101.jpg
fear/occ_img_97.jpg
fear/occ_img_117.jpg
fear/occ_img_72.jpg
fear/occ_img_70.jpg
fear/occ_img_111.jpg
fear/occ_img_105.jpg
fear/occ_img_91.jpg
fear/occ_img_104.jpg
fear/occ_img_79.jpg
fear/occ_img_108.jpg
anger/occ_img_134.jpg
anger/occ_img_123.jpg
anger/occ_img_152.jpg
anger/occ_img_166.jpg
happy/occ_img_198.jpg
happy/occ_img_182.jpg
happy/occ_img_220.jpg
happy/occ_img_199.jpg
happy/occ_img_208.jpg
happy/occ_img_229.jpg
happy/occ_img_221.jpg
happy/occ_img_188.jpg
happy/occ_img_236.jpg
surprise/occ_img_248.jpg
surprise/occ_img_259.jpg
surprise/occ_img_276.jpg
surprise/occ_img_273.jpg
surprise/occ_img_260.jpg
surprise/occ_img_244.jpg
sad/occ_img_309.jpg
sad/occ_img_366.jpg
sad/occ_img_358.jpg
sad/occ_img_313.jpg
sad/occ_img_333.jpg
sad/occ_img_302.jpg
neural/occ_img_379.jpg
neural/occ_img_382.jpg
neural/occ_img_395.jpg
neural/occ_img_402.jpg
neural/occ_img_405.jpg
neural/occ_img_389.jpg
neural/occ_img_433.jpg
neural/occ_img_412.jpg
neural/occ_img_381.jpg
neural/occ_img_436.jpg
neural/occ_img_399.jpg
neural/occ_img_403.jpg
neural/occ_img_396.jpg
neural/occ_img_421.jpg
neural/occ_img_384.jpg

Will be grateful if you could share those images.

Thanks

How to run

Can you please explain step by step on how to run and train the model
Thanks in advance

Question about training pACNN

I follow the instruction in paper.
I have concern about unobstructed regions in facial image(24 regions). Can you tell me where the code implement it, if i have 24 points how to crop region in each point is good and how to train ??? In pACNN whether to get 24 points to train? Thank in advance for your help.

有关训练的问题

您好,我已经编译成功了,请问怎么训练已有的数据呢,24个点的训练数据也准备好了,谢谢!

Ask question about 10-fold subject-independent cross validation

How do you conduct the 10-fold subject-independent cross validation on CK+ database? Do you divide 118 persons into 10 fold and take 106 persons' expressions as training set, the rest 12 persons' expressions as testing set? Do you save the best test accuracy in each fold and average them as the final result?

你是如何在CK+数据库上进行10折交叉验证实验的呢?你是把118个人平均分为10份,每次轮流取106个人的全部表情作为训练集,剩下12个人的全部表情作为测试集吗?你最后的准确率是怎么计算的呢?是取每一折的收敛时候的准确率来计算,还是取最好的准确率来计算最后的平均准确率呢?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.