On the use of Action Units and fuzzy explanatory models for facial expression recognition

Abstract

Facial expression recognition is related to the automatic identification of affective states of a subject by computational means. Facial expression recognition is used for many applications, such as security, human-computer interaction, driver safety, and health care. Although many works aim to tackle the problem of facial expression recognition, and the discriminative power may be acceptable, current solutions have limited explicative power, which is insufficient for certain applications, such as facial rehabilitation. Our aim is to alleviate the current limited explicative power by exploiting explainable fuzzy models over sequences of frontal face images. The proposed model uses appearance features to describe facial expressions in terms of facial movements, giving a detailed explanation of what movements are in the face, and why the model is making a decision. The model architecture was selected to keep the semantic meaning of the found facial movements. The proposed model can discriminate between the seven basic facial expressions, obtaining an average accuracy of 90.8±14%, with a maximum value of 92.9±28%.

Effects of lip and eye color on the emotion categorization of facial expressions

“…In the present study, we focused on low‐level features to detect color effects. Previous research focused on dynamic or static approaches according to the Facial Action Coding System (FACS), which describes each basic facial expression with small facial movements called “action units” (AUs) 31 . By contrast, we conducted this experiment based on the hypothesis that perception of emotional valence in facial expressions was not only affected by the facial movements themselves (supported by the results of comparing two different facial features) but also by other variables, such as color or the combination of color and facial features.…”

Graph Convolution with Low-rank Learnable Local Filters

“…While CNN methods [14,19,36] prevail in FER task, landmark methods have the potential advantage in lighter model size as well as more robustness to pose variation. Earlier methods based on facial landmarks used handcrafted features [20,38] rather than deep networks. Skeleton-based methods in action recognition have been developed intensively recently [44], including non-deep methods [49,50] and deep methods [23,25,30,53].…”

“…Another class of applications consider learning representation of graph signals x i : V → R on a fixed or mostly fixed underlying graph G = (V, E), that is, to classify {A, x i } → y i . Examples include spherical mesh data [10,11,15,21], data on manifolds in computer graphics [3,16,35,37], and landmark data on human face and body [20,23,25,30,38,44,49,50,53], the last one being a primary motivating application of our work. The problem relates to convolutional neural network (CNN) on non-Euclidean domain [4], and a challenge lies in that mesh can be irregular and coarse, e.g., the body landmarks in action recognition.…”