Psychometric challenges and proposed solutions when scoring facial emotion expression codes

Abstract

Coding of facial emotion expressions is increasingly performed by automated emotion expression scoring software; however, there is limited discussion on how best to score the resulting codes. We present a discussion of facial emotion expression theories and a review of contemporary emotion expression coding methodology. We highlight methodological challenges pertinent to scoring software-coded facial emotion expression codes and present important psychometric research questions centered on comparing competing scoring procedures of these codes. Then, on the basis of a time series data set collected to assess individual differences in facial emotion expression ability, we derive, apply, and evaluate several statistical procedures, including four scoring methods and four data treatments, to score software-coded emotion expression data. These scoring procedures are illustrated to inform analysis decisions pertaining to the scoring and data treatment of other emotion expression questions and under different experimental circumstances. Overall, we found applying loess smoothing and controlling for baseline facial emotion expression and facial plasticity are recommended methods of data treatment. When scoring facial emotion expression ability, maximum score is preferred. Finally, we discuss the scoring methods and data treatments in the larger context of emotion expression research.

Human Observers and Automated Assessment of Dynamic Emotional Facial Expressions: KDEF-dyn Database Validation

“…Nevertheless, some expressions, especially, happiness, and also disgust and surprise, were classified better than sadness, anger, and fear (see Table 2 ), which is in total agreement with results obtained with other automated computation algorithms ( Lucey et al, 2010 ). Second, AUs generally discriminated between expressive categories, and this was in accordance with FACS proposals ( Ekman et al, 2002 ; Olderbak et al, 2014 ). Some AUs characterized expressions more specifically or strongly than others (see Table 3 ), e.g., AU12 for happiness, AU25 for surprise, AUs 9 and 10 for disgust, AU1 for fear, and AU4 for anger and sadness (the AU4 combination with other AUs allowed for a clear discrimination between these two expressions; see Table 2 ).…”

“…Recently, FACET has been used in psychological and applied research (see Dente et al, 2017 ). The automated analysis provides two types of measures (see Gordon et al, 2011 ; Olderbak et al, 2014 ): (a) expression evidence scores for each category: joy, anger, surprise, fear, disgust, sadness, and contempt, in addition to neutral; and (b) AUs evidence scores (for 20 AUs: 1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 18, 20, 23, 24, 25, 26, 28, and 43), according to FACS ( Ekman et al, 2002 ); see also ( Cohn et al, 2007 ; Cohn and De la Torre, 2015 ). AUs are anatomically related to the movement of specific face muscles (e.g., AU12 involves the contraction of the zygomaticus major muscle, which draws the angle of the mouth superiorly and posteriorly to allow for smiling).…”

“…First, we collected data from human observers in an expression categorization task including measures of (a) correct recognition responses, i.e., the probability that they coincided with the intended KDEF expression, (b) reaction times indicating processing efficiency, and (c) the probability of confusions across different expressions, for each of the six basic emotions. Second, with Emotient FACET software (v. 6.1.2667.3; iMotions), we performed automated facial expression analyses ( Bartlett and Whitehill, 2011 ; Olderbak et al, 2014 ; Cohn and De la Torre, 2015 ; Girard et al, 2015 ; Dente et al, 2017 ) of (a) the probability of each expression to be detected, as a function of spatial maps of facial features, and also (b) the probability of each of 20 AUs to be activated, i.e., muscle movements, according to FACS ( Ekman and Friesen, 1978 ; Ekman et al, 2002 ). The automated analyses of expressions and AUs were performed for 31 intensities (including the neutral baseline) of each emotional facial expression (including apex), while the human recognition measures were obtained for the maximum expressive intensity only.…”

Association between facial expression and PTSD symptoms among young children exposed to the Great East Japan Earthquake: a pilot study

“…This technology has potential applications in research involving PTSD among both children and adults. Previous studies using emotional facial expression software were limited to adults ( Samal and Iyengar, 1992 ; Zeng et al, 2009 ; Olderbak et al, 2014 ). Prior studies have not used the software with child study participants.…”

The Neurobiology of Dispositional Negativity and Attentional Biases to Threat: Implications for Understanding Anxiety Disorders in Adults and Youth

“…J. Wilson, Smyth, & MacLean, 2014). The development of robust mobile eye trackers (e.g., Applied Science Laboratories’ Mobile Eye system), the emergence of commercial software for automated facial analytics (e.g., from Affectiva, Emotient, and Noldus; Olderbak, Hildebrandt, Pinkpank, Sommer, & Wilhelm, 2014), and the widespread dissemination of smart phone technology afford additional opportunities for objectively and unobtrusively quantifying social attention, context, and daily behavior (Gosling & Mason, 2015; Sano et al, 2015; Wrzus & Mehl, 2015). Combining these measures with laboratory assays of brain function would open the door to discovering the neural systems underlying maladaptive experiences and pathology-promoting behaviors (e.g., social withdrawal, avoidance, and hyper-vigilance) in the real world, close to clinical end-point (Price et al, 2016).…”