Explaining Facial Expression Recognition

Facial expression recognition (FER) has emerged as a promising approach to the development of emotion-aware intelligent agents and systems. However, key challenges remain in utilizing FER in real-world contexts, including ensuring user understanding and establishing a suitable level of user trust. We developed a novel explanation method utilizing Facial Action Units (FAUs) to explain the output of a FER model through both textual and visual modalities. We conducted an empirical user study evaluating user understanding and trust, comparing our approach to stateof-the-art eXplainable AI (XAI) methods. Our results indicate that visual AND textual as well as textual-only FAU-based explanations resulted in better user understanding of the FER model. We also show that all modalities of FAU-based methods improved appropriate trust of the users towards the FER model.

Authors

• Sanjeev Nahulanthran Monash University
• Leimin Tian CSIRO Robotics
• Dana Kulic
• Mor Vered Monash University

Keywords

• Facial Expression Recognition
• Explainability
• Transparency
• Trust
• Facial Action Unit