Real-time inference of mental states from facial expressions and upper body gestures

Abstract

Abstract-We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1,2,4,6,7,10,12,15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.

A feasibility study in using facial expressions analysis to evaluate player experiences

“…They generally reported a high level of correlation between the system's findings and human expert analyses. In other domains, general emotion detection based on facial expression recognition [12,1] have also shown promising results.…”

Inferring Player Experiences Using Facial Expressions Analysis

“…For example, Branco [2] showed some encouraging results evaluating positive and negative expressions of users of an online shopping website. In other domains, general emotion detection based on facial expression recognition [1,10] have also shown promising results.…”

“…These are derived from the six universal expressions that has been shown to be a basis for emotions across diverse cultures [7]. The use of these six basic expressions, as opposed to the more detailed Facial Action Coding System (FACS) [8] is a conscious decision due to the fact that FACS action unit recognition being still an open problem [1,13]. In the future, we will gradually investigate the feasibility of using FACS as action unit recognition improves.…”

Frame-Based Facial Expression Recognition Using Geometrical Features

“…They used IR illuminations and Kalman filtering to assist the facial point detection and tracking. Baltrušaitis et al [5] proposed a dynamic system with three levels of inference on progressively longer time scales to understand the human mental states from facial expressions and upper-body gestures, where they employed both DBN and HMM. Lörincz et al [19] used time-series kernels to analyze the spatiotemporal process of the facial points, where the points' movements in 3D space are classified with kernels derived from time-warping similarity measures.…”

“…The human mental state could be inferred using various modalities such as facial expressions, hand gestures, acoustic data, and biophysiological data [1][2][3][4][5]. The importance of knowing this mental state appears in different disciplines.…”