Emotion recognition from posed and spontaneous dynamic expressions: Human observers versus machine analysis.

Krumhuber, Eva; Küster, Dennis; Namba, Shushi; Shah, Datin; Calvo, Manuel G.

doi:10.1037/emo0000712

Cited by 52 publications

(49 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We opted for this response format to allow for direct comparability with the automatic classifiers' recognition data using pre-specified emotion labels. As shown in prior research, adding a no/other emotion escape option does not change the overall level of target emotion recognition [51]. Instead, it only prevents agreement on incorrect labels when the target emotion label is absent [52].…”

Section: Human Observersmentioning

confidence: 68%

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

et al. 2020

Self Cite

View full text Add to dashboard Cite

In the wake of rapid advances in automatic affect analysis, commercial automatic classifiers for facial affect recognition have attracted considerable attention in recent years. While several options now exist to analyze dynamic video data, less is known about the relative performance of these classifiers, in particular when facial expressions are spontaneous rather than posed. In the present work, we tested eight out-of-the-box automatic classifiers, and compared their emotion recognition performance to that of human observers. A total of 937 videos were sampled from two large databases that conveyed the basic six emotions (happiness, sadness, anger, fear, surprise, and disgust) either in posed (BU-4DFE) or spontaneous (UT-Dallas) form. Results revealed a recognition advantage for human observers over automatic classification. Among the eight classifiers, there was considerable variance in recognition accuracy ranging from 48% to 62%. Subsequent analyses per type of expression revealed that performance by the two best performing classifiers approximated those of human observers, suggesting high agreement for posed expressions. However, classification accuracy was consistently lower (although above chance level) for spontaneous affective behavior. The findings indicate potential shortcomings of existing out-of-thebox classifiers for measuring emotions, and highlight the need for more spontaneous facial databases that can act as a benchmark in the training and testing of automatic emotion recognition systems. We further discuss some limitations of analyzing facial expressions that have been recorded in controlled environments.

show abstract

Section: Human Observersmentioning

confidence: 68%

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Those typically involved posed or acted facial behavior displaying prototypical patterns of emotional expression. In this vein, machine classification performance was found to be high for deliberately posed stimuli (Beringer et al, 2019;Skiendziel et al, 2019), but was reduced when facial expressions were spontaneous and/or subtle in their appearance (Yitzhak et al, 2017;Krumhuber et al, 2020). Unless training sets encompass large stimulus collections, automatic systems may therefore fail to generalize to the wide variety of expressive displays common in everyday life.…”

Section: Introductionmentioning

confidence: 99%

Human and machine validation of 14 databases of dynamic facial expressions

et al. 2020

Self Cite

View full text Add to dashboard Cite

With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli have been developed over the past two decades. The present research aimed to test existing sets of dynamic facial expressions (published between 2000 and 2015) in a cross-corpus validation effort. For this, 14 dynamic databases were selected that featured facial expressions of the basic six emotions (anger, disgust, fear, happiness, sadness, surprise) in posed or spontaneous form. In Study 1, a subset of stimuli from each database (N = 162) were presented to human observers and machine analysis, yielding considerable variance in emotion recognition performance across the databases. Classification accuracy further varied with perceived intensity and naturalness of the displays, with posed expressions being judged more accurately and as intense, but less natural compared to spontaneous ones. Study 2 aimed for a full validation of the 14 databases by subjecting the entire stimulus set (N = 3812) to machine analysis. A FACS-based Action Unit (AU) analysis revealed that facial AU configurations were more prototypical in posed than spontaneous expressions. The prototypicality of an expression in turn predicted emotion classification accuracy, with higher performance observed for more prototypical facial behavior. Furthermore, technical features of each database (i.e., duration, face box size, head rotation, and motion) had a significant impact on recognition accuracy. Together, the findings suggest that existing databases vary in their ability to signal specific emotions, thereby facing a trade-off between realism and ecological validity on the one end, and expression uniformity and comparability on the other.

show abstract

“…Given the large and growing number of choices for academics and practitioners in consumer research, there still exists little "cross-system" (i.e., between competing products) validation research that could independently inform about the relative performance indicators of AHAA (Krumhuber et al, 2019). Out of the studies available to date, only a few have directly compared different commercial classifiers (Stöckli et al, 2018).…”

Section: Abundant Choices: Classifiers Lack Cross-system Validationmentioning

confidence: 99%

“…Out of the studies available to date, only a few have directly compared different commercial classifiers (Stöckli et al, 2018). Likewise, a small number of studies has tested AHAA against human performance benchmarks on a larger number of databases (Yitzhak et al, 2017;Krumhuber et al, 2019), thereby calling the generalizability of findings derived from single stimulus sets into question.…”

Section: Abundant Choices: Classifiers Lack Cross-system Validationmentioning

confidence: 99%

“…Generalizability of AHAA study findings may be further limited in other ways. E.g., classifier performance may be substantially lower for spontaneous affective behavior (Dupré et al, 2019;Krumhuber et al, 2019). This issue often ties into the lack of information given about the stimulus materials originally used to develop or "train" a given AHAA system to fully evaluate generalizability toward similar novel stimuli.…”

Section: Abundant Choices: Classifiers Lack Cross-system Validationmentioning

confidence: 99%

See 1 more Smart Citation

Opportunities and Challenges for Using Automatic Human Affect Analysis in Consumer Research

et al. 2020

Self Cite

View full text Add to dashboard Cite

The ability to automatically assess emotional responses via contact-free video recording taps into a rapidly growing market aimed at predicting consumer choices. If consumer attention and engagement are measurable in a reliable and accessible manner, relevant marketing decisions could be informed by objective data. Although significant advances have been made in automatic affect recognition, several practical and theoretical issues remain largely unresolved. These concern the lack of cross-system validation, a historical emphasis of posed over spontaneous expressions, as well as more fundamental issues regarding the weak association between subjective experience and facial expressions. To address these limitations, the present paper argues that extant commercial and free facial expression classifiers should be rigorously validated in cross-system research. Furthermore, academics and practitioners must better leverage fine-grained emotional response dynamics, with stronger emphasis on understanding naturally occurring spontaneous expressions, and in naturalistic choice settings. We posit that applied consumer research might be better situated to examine facial behavior in socio-emotional contexts rather than decontextualized, laboratory studies, and highlight how AHAA can be successfully employed in this context. Also, facial activity should be considered less as a single outcome variable, and more as a starting point for further analyses. Implications of this approach and potential obstacles that need to be overcome are discussed within the context of consumer research.

show abstract

Emotion recognition from posed and spontaneous dynamic expressions: Human observers versus machine analysis.

Cited by 52 publications

References 26 publications

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

Human and machine validation of 14 databases of dynamic facial expressions

Opportunities and Challenges for Using Automatic Human Affect Analysis in Consumer Research

Contact Info

Product

Resources

About