Using computer-vision and machine learning to automate facial coding of positive and negative affect intensity

Abstract

Facial expressions are fundamental to interpersonal communication, including social interaction, and allow people of different ages, cultures, and languages to quickly and reliably convey emotional information. Historically, facial expression research has followed from discrete emotion theories, which posit a limited number of distinct affective states that are represented with specific patterns of facial action. Much less work has focused on dimensional features of emotion, particularly positive and negative affect intensity. This is likely, in part, because achieving inter-rater reliability for facial action and affect intensity ratings is painstaking and labor-intensive. We use computer-vision and machine learning (CVML) to identify patterns of facial actions in 4,648 video recordings of 125 human participants, which show strong correspondences to positive and negative affect intensity ratings obtained from highly trained coders. Our results show that CVML can both (1) determine the importance of different facial actions that human coders use to derive positive and negative affective ratings when combined with interpretable machine learning methods, and (2) efficiently automate positive and negative affect intensity coding on large facial expression databases. Further, we show that CVML can be applied to individual human judges to infer which facial actions they use to generate perceptual emotion ratings from facial expressions.

Negative Affect Induces Rapid Learning of Counterfactual Representations: A Model-based Facial Expression Analysis Approach

“…To our knowledge, this is the first study of its kind to include dynamic facial expressions as direct input into a cognitive model, although similar model-based approaches are becoming increasingly common in cognitive neuroscience (Turner, Forstmann, Love, Palmeri, & Van Maanen, 2017). Further, work using automated facial expression coding is gaining traction in social and behavioral sciences due to its efficiency relative to human coders (e.g., Cheong, Brooks, & Chang, 2017;Haines et al, 2019). Future work would benefit from combining automated facial expression coding with behavioral paradigms that collect self-reports of emotion (e.g., Rutledge et al, 2014), which would both allow for more strenuous validity tests of automated measures and create opportunities for exploring the relationships between unobservable and observable emotional states.…”

“…The AFEC model first uses FACET-a computer vision software (iMotions, 2018)-to detect the presence of 20 different facial action units (Ekman, Friesen, & Hager, 2002), which are then translated to affect intensity ratings using a machine learning model that we previously validated. In our validation study, the model showed correlations with human observer ratings of .89 and .76 for positive and negative affect intensity, respectively (for more details, see Haines et al, 2019). Figure 5 shows the steps used to preprocess and apply the AFEC model to our participants' facial expressions in response to outcome feedback.…”

“…Here, we use: (1) risky decision-making tasks that evoke counterfactual thinking; 2computational, emotion-driven reinforcement learning models built on regret and disappointment theory; and (3) automated computer-vision coding of facial affect intensity (e.g., Haines et al, 2019) Specifically, we develop models within the framework of traditional regret and disappointment theories (e.g., Ahn et al, 2012;Delquié & Cillo, 2006;Loomes & Sugden, 1982;), which allows us to test assumptions underlying functional theories of counterfactual thinking. We test a novel model assuming that that individuals learn a representation of counterfactual outcomes within the choice environment as they gain experience with each outcome, which provides a bridge between both: (1) traditional expected utility models and counterfactual models, and (2) description-and experience-based task formats that offer feedback.…”

“…We excluded any trials where a participant's face was detected for less than 10% of the total 30 samples in the given trial (~3% of trials excluded in total). After iterating step 2 for each trial and participant (step 3), we entered the resulting values as predictors in the AFEC model described above to generate intensity ratings for positive and negative affect (Haines et al, 2019). We used the positive and negative affect ratings as input into the computational models as described below.…”

Read My Face: Automatic Facial Coding Versus Psychophysiological Indicators of Emotional Valence and Arousal

“…In one preliminary study only a small set of pictures -three pleasant and unpleasant emotional scenes -were used to elicit facial responses with moderate to good classification performance on a categorical analysis level (Stöckli et al, 2018). The other study demonstrated good prediction of unpleasant versus pleasant facial responses with an AU-based machine learning procedure (Haines et al, 2019). Unfortunately, in both studies there was no neutral picture category as a comparative condition.…”