Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework

Abstract

Affect recognition based on subjects' facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This paper exploits these advances and presents significant contributions for affect analysis and recognition in-the-wild. Affect analysis and recognition can be seen as a dual knowledge generation problem, involving: i) creation of new, large and rich in-the-wild databases and ii) design and training of novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. The paper focuses on large in-the-wild databases, i.e., Aff-Wild and Aff-Wild2 and presents the design of two classes of deep neural networks trained with these databases. The first class refers to uni-task affect recognition, focusing on prediction of the valence and arousal dimensional variables. The second class refers to estimation of all main behavior tasks, i.e. valence-arousal prediction; categorical emotion classification in seven basic facial expressions; facial Action Unit detection. A novel multi-task and holistic framework is presented which is able to jointly learn and effectively generalize and perform affect recognition over all existing in-the-wild databases. Large experimental studies illustrate the achieved performance improvement over the existing state-of-the-art in affect recognition.

An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild

“…They also proposed multitask learning models that employ both visual and audio modalities and suggested to use the ArcFace loss [6] for expression recognition. In an additional work, Kollias et al [21] [18] studied the problem of non-overlapping annotations in multitask learning datasets. They explored task-relatedness and proposed a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions.…”

Privileged Information for Modeling Affect In The Wild

“…They demonstrate that multimodal deep learning affect models can significantly improve affect detection in the wild. Finally, Kolias and Zafeiriou [34] propose a unified framework for affect modeling in the wild that considers facial expressions and categorical affect, facial action units, and dimensional affect representations.…”

Causal affect prediction model using a facial image sequence

“…However, there are many obstacles for HCI systems used in real-world applications, such as in-the-wild or real-time tasks. To address these problems, Kollias et al have been hosting the Affective Behavior Analysis in-the-wild (ABAW) Competition, which involves a variety of research activites, for two years [5,6,7,8,9,11,12,28]. Most of the top-ranked teams in the first challenge of ABAW (ABAW1) [6], held in conjunction with the 15 th IEEE Conference on Face and Gesture Recognition (FG2020), used convolutional neural networks (CNNs) with single facial images or sequences of such images.…”