AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition

Recognizing human emotion/expressions automatically is quite an expected ability for intelligent robotics, as it can promote better communication and cooperation with humans. Current deep-learning-based algorithms may achieve impressive performance in some lab-controlled environments, but they always fail to recognize the expressions accurately for the uncontrolled in-the-wild situation. Fortunately, facial action units (AU) describe subtle facial behaviors, and they can help distinguish uncertain and ambiguous expressions. In this work, we explore the correlations among the action units and facial expressions, and devise an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. Specifically, it leverages AU-expression correlations to guide the learning of the AU classifiers, and thus it can obtain AU representations without incurring any AU annotations. Then, it introduces a knowledge-guided attention mechanism that mines useful AU representations under the constraint of AU-expression correlations. In this way, the framework can capture local discriminative and complementary features to enhance facial representation for facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods. Codes and trained models are available at https://github.com/HCPLab-SYSU/AUE-CRL.

Authors

• Tao Pu 0002
• Tianshui Chen
• Yuan Xie 0004
• Hefeng Wu
• Liang Lin

Keywords

• Gold
• Emotion recognition
• Correlation
• Codes
• Automation
• Annotations
• Face recognition
• Facial Expressions
• Representation Learning
• Facial Expression Recognition
• Learning Framework
• Attention Mechanism
• Discriminative Features
• Action Units
• Ambiguous Expressions
• Facial Action Units
• Neural Network
• Deep Learning
• Convolutional Network
• Convolutional Neural Network
• Deep Neural Network
• Feature Maps
• Average Accuracy
• Disgust
• Stochastic Gradient Descent
• Current Dataset
• Pseudo Labels
• Feature Map Size
• Mean Square Error Loss
• Conditional Random Field
• Face Images
• Histogram Of Oriented Gradients
• Facial Action Coding System
• Hyperbolic Tangent
• Balance Parameters
• Additional Annotations