Training Emotion Recognition Accuracy: Results for Multimodal Expressions and Facial Micro Expressions

Abstract

Nonverbal emotion recognition accuracy (ERA) is a central feature of successful communication and interaction, and is of importance for many professions. We developed and evaluated two ERA training programs—one focusing on dynamic multimodal expressions (audio, video, audio-video) and one focusing on facial micro expressions. Sixty-seven subjects were randomized to one of two experimental groups (multimodal, micro expression) or an active control group (emotional working memory task). Participants trained once weekly with a brief computerized training program for three consecutive weeks. Pre-post outcome measures consisted of a multimodal ERA task, a micro expression recognition task, and a task about patients' emotional cues. Post measurement took place approximately a week after the last training session. Non-parametric mixed analyses of variance using the Aligned Rank Transform were used to evaluate the effectiveness of the training programs. Results showed that multimodal training was significantly more effective in improving multimodal ERA compared to micro expression training or the control training; and the micro expression training was significantly more effective in improving micro expression ERA compared to the other two training conditions. Both pre-post effects can be interpreted as large. No group differences were found for the outcome measure about recognizing patients' emotion cues. There were no transfer effects of the training programs, meaning that participants only improved significantly for the specific facet of ERA that they had trained on. Further, low baseline ERA was associated with larger ERA improvements. Results are discussed with regard to methodological and conceptual aspects, and practical implications and future directions are explored.

Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

“…Other factors also influence this task. Döllinger et al (2021) review them, pointing to peer status and friendship quality (Wang et al 2019), to the possible state of depression of the observers (Dalili et al 2015), and to their personality traits (Hall, Mast, and West 2016) -conscientiousness, and openness are positively correlated to the ability to recognize nonverbal expressions of emotions, while shyness and neuroticism are negatively associated with it (Hall, Mast, and West 2016). We also assess personality traits and state-specific variables in our study.…”

An ingroup disadvantage in recognizing micro-expressions

“…In both the pre-test and post-test of micro-expression recognition tasks, we employed the well-accepted paradigm of Japanese and Caucasian Brief Affect Recognition Test (JACBART; e.g., Matsumoto et al, 2000;Hurley, 2012;Hurley et al, 2014;Wu et al, 2016aWu et al, , 2022Demetrioff et al, 2017;Zhang et al, 2020;Döllinger et al, 2021;Fan et al, 2022) to present microexpressions. In this paradigm, a fixation cross was presented for 500 ms at first, after that a facial expression image was presented according to its assigned duration (i.e., 100 ms or 333 ms in Study 1A), which was sandwiched in between two 1 s presentations of the same expresser's neutral face (see Figure 1).…”

“…As far as we know, no previous studies have directly addressed this issue. Answers to this question will deepen our understanding of the micro-expression recognition process which provides the necessary knowledge for developing more efficient micro-expression recognition training programs (Matsumoto and Hwang, 2011;Hurley et al, 2014;Döllinger et al, 2021) or building more valid micro-expression databases (Yan et al, 2014;Yap et al, 2020;Li et al, 2022). In fact, previous studies have found that microexpressions and macro-expressions are usually identical in their appearances (Ekman, 2003;Hwang, 2011, 2018;Yan et al, 2013) and there are some similarities between the recognition of micro-expressions and the recognition of macro-expressions.…”

“…In fact, previous studies have found that microexpressions and macro-expressions are usually identical in their appearances (Ekman, 2003;Hwang, 2011, 2018;Yan et al, 2013) and there are some similarities between the recognition of micro-expressions and the recognition of macro-expressions. For example, researchers have found that both the recognitions of micro-expression and macro-expression are affected by factors like age, personality, profession, training experience, mental disorders of perceivers, and the emotional context of facial expressions (Matsumoto et al, 2000;Frank et al, 2009;Hurley, 2012;Hurley et al, 2014;Svetieva and Frank, 2016;Demetrioff et al, 2017;Zhu et al, 2017;Zhang et al, 2020;Döllinger et al, 2021;Fan et al, 2022). However, previous studies also suggest that there might be fundamental differences in the neuropsychological basis for micro-expression recognition and macro-expression recognition.…”

Oxytocin Impairs the Recognition of Micro-Expressions of Surprise and Disgust

“…The short duration of micro-expression is not the only obstacle we encountered during micro-expression recognition. To the best of our knowledge, only a few studies have tried to investigate the psychological or brain mechanisms of micro-expression recognition, which makes the scientists difficult to design efficient micro-expression recognition training programs (e.g., Ekman, 2002 ; Frank et al, 2009 ; Matsumoto and Hwang, 2011 ; Döllinger et al, 2021 ). For example, researchers have found that the factors like emotional context, age, childhood family environment, personality, and profession (e.g., Frank et al, 2009 ; Hurley et al, 2014 ; Zhang et al, 2014 , 2020b ; Svetieva and Frank, 2016 ; Demetrioff et al, 2017 ; Felisberti, 2018 ) may affect the recognition accuracy of micro-expressions.…”