Crossing Domains for AU Coding: Perspectives, Approaches, and Measures

Abstract

Facial action unit (AU) detectors have performed well when trained and tested within the same domain. How well do AU detectors transfer to domains in which they have not been trained? We review literature on cross-domain transfer and conduct experiments to address limitations of prior research. We evaluate generalizability in four publicly available databases. EB + (an expanded version of BP4D + ), Sayette GFT, DISFA and UNBC Shoulder Pain (SP). The databases differ in observational scenarios, context, participant diversity, range of head pose, video resolution, and AU base rates. In most cases performance decreased with change in domain, often to below the threshold needed for behavioral research. However, exceptions were noted. Deep and shallow approaches generally performed similarly and average results were slightly better for deep model compared to shallow one. Occlusion sensitivity maps revealed that local specificity was greater for AU detection within than cross domains. The findings suggest that more varied domains and deep learning approaches may be better suited for generalizability and suggest the need for more attention to characteristics that vary between domains. Until further improvement is realized, caution is warranted when applying AU classifiers from one domain to another.

Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases

“…Because manual FACS coding has the disadvantage of being time consuming, automatic detection of FACS AUs has been an active area of research [ 5 ]. Automated facial AU detection systems are available as both commercial tools (e.g., Affectiva, FaceReader) and open-source tools (e.g., OpenFace, [ 6 , 7 ] and Automated Facial Affect Recognition (AFAR) [ 8 , 9 ]).…”

“…Automated facial AU detection systems are available as both commercial tools (e.g., Affectiva, FaceReader) and open-source tools (e.g., OpenFace, [ 6 , 7 ] and Automated Facial Affect Recognition (AFAR) [ 8 , 9 ]). One study found that OpenFace and AFAR generally performed similarly, but average results were slightly better for AFAR [ 5 ]. In addition, the performance of automatic AU detection by the current commercial systems has been checked in many ways.…”

“…The current study also hypothesized that all machines would show better AU detection performance with posed rather than with spontaneous facial expressions. Although there might be little difference in performance between OpenFace and AFAR [ 5 ], we had no directional hypothesis for the commercial software. As for the machines’ bias scores, we evaluated each system systematically and exploratively.…”

Top Ranked Abstracts from the 2021 Annual Meeting of the Society for Affective Science

“…Methods Sixty-one 4-month-olds and their mothers completed two minutes of face-to-face interaction. Separate mother and infant synchronized video-recordings were submitted to automated facial affect recognition (AFAR; Ertuğrul et al, 2019;Ertuğrul et al, 2020). The AFAR computer vision software uses a fast-cascade regression framework tracked and normalized facial images from which the degree of mouth opening was calculated.…”

Systematic Evaluation of Design Choices for Deep Facial Action Coding Across Pose

“…To meet the need for systems that are robust to new contexts, systems must perform well both in the domains from which they come and in the domains to which they may be applied. The evaluation of domain transfer in AU systems is relatively new (Cohn et al, 2019;Ertugrul et al, 2020).…”