Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Abstract

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions.

What’s in a face: Automatic facial coding of untrained study participants compared to standardized inventories

“…In this study we directly compared state-of-the-art Automatic Facial Coding (AFC) measures of emotional facial expressions generated by untrained participants in a typical laboratory setting and prototypical facial expressions from standardized inventories (i.e., trained actors). Untrained participants compared to trained actors showed substantially less intense facial expressions which is in line with previous research [ 26 , 27 ]. Our present study indicates that most emotion categories, in particular joyful faces, can be detected with both high sensitivity and specificity.…”

“…In fact, prototypical facial expressions (i.e., expressions of trained actors in the present study) are recognized by AFC much more clearly than more naturalistic emotional facial expressions [ 27 ]. However, AFC accuracy of such prototypical facial expressions does not directly correspond with accuracy of analyses in naturally occurring emotional facial reactions.…”

“…Machine learning procedures underlying the AFC software were trained with emotional facial expressions from standardized inventories, they may therefore be best suited to classify prototypical facial expressions. When untrained study participants do not display prototypical facial expressions or display them with less intensity [ 26 , 27 ] these algorithms may not be as successful.…”

Relationships between Nursing Students’ Skill Mastery, Test Anxiety, Self-Efficacy, and Facial Expressions: A Preliminary Observational Study

“…Non-standardised images stem from more natural settings (i.e., stills from movie scenes that display emotions in the actors’ faces) [ 23 ]. Azure showed superior performance on non-standardised facial expressions among other facial expression recognition tools [ 24 ] that performed facial expression analysis. When we input one snapshot in the application, it showed the facial emotion analysis and saved the data to an Excel Spreadsheet (Microsoft, Redmond, WA, USA).…”

Dataset on individual differences in self-reported personality and inferred emotional expression in profile pictures of Italian Facebook users

“…The code used for performing the API calls was adapted from the code provided by Theresa Küntzler (University of Konstanz) on GitHub [12] as used in a previous publication (e.g. [13] ).…”