Test–Retest Reliability in Automated Emotional Facial Expression Analysis: Exploring FaceReader 8.0 on Data from Typically Developing Children and Children with Autism

Abstract

Automated emotional facial expression analysis (AEFEA) is used widely in applied research, including the development of screening/diagnostic systems for atypical human neurodevelopmental conditions. The validity of AEFEA systems has been systematically studied, but their test–retest reliability has not been researched thus far. We explored the test–retest reliability of a specific AEFEA software, Noldus FaceReader 8.0 (FR8; by Noldus Information Technology). We collected intensity estimates for 8 repeated emotions through FR8 from facial video recordings of 60 children: 31 typically developing children and 29 children with autism spectrum disorder. Test–retest reliability was imperfect in 20% of cases, affecting a substantial proportion of data points; however, the test–retest differences were small. This shows that the test–retest reliability of FR8 is high but not perfect. A proportion of cases which initially failed to show perfect test–retest reliability reached it in a subsequent analysis by FR8. This suggests that repeated analyses by FR8 can, in some cases, lead to the “stabilization” of emotion intensity datasets. Under ANOVA, the test–retest differences did not influence the pattern of cross-emotion and cross-group effects and interactions. Our study does not question the validity of previous results gained by AEFEA technology, but it shows that further exploration of the test–retest reliability of AEFEA systems is desirable.

Emotional contagion in dyadic online video conferences - Empirical evidence based on self-report and facial expression data [PREPRINT]

“…Instead, we used automated facial expression analysis to measure changes in the outward appearance of an individual's face, while not necessarily coinciding with an individual's subjective experiences, to explore the degree to which those expressions are transmitted even across the "obstacle" of a digitally mediated social interaction. Further, recent validation studies (see Dupré et al, 2020 for an overview) reported that automatic facial expression analysis algorithms offer sufficiently high reliability in comparison to human FACS coders (Höfling et al, 2022;Küntzler et al, 2021;Skiendziel et al, 2019;Stöckli et al, 2018) and to itself when measured at two measurement points (Borsos et al, 2022). In that sense, we conclude that facial expressions -independent of the underlying emotional state -represent highly relevant nonverbal cues and a channel for the interpersonal communication of subjective emotional experiences that can be reliably assessed using automated facial expression analysis.…”

The nonverbal expression of guilt in healthy adults

“…Recordings with a resolution of 640 × 480 at 33 frames per second were saved as MP4 files and analyzed frame by frame by FaceReader 8.1, an automated facial expression analysis software, a reliable and valid method for analyzing facial expression data (Noldus Information Technology, Wageningen, The Netherlands) 39 , 40 . FaceReader was set to detect each of the twenty most common facial Action Units (AUs), which were reported on a scale between 0 (not present) and 1 (maximally present).…”

API Face Value

“…From a measurement perspective, some studies have raised concern about the reliability of the software's emotion estimates. Borsos et al (2022) evaluated the test-retest reliability of emotion API and found small but significant differences in the ratings. Flynn et al (2020) observed group differences in the accuracy of emotion estimates between children and adults.…”