Facial Affect “In-the-Wild”: A Survey and a New Database

Abstract

Well-established databases and benchmarks have been developed in the past 20 years for automatic facial behaviour analysis. Nevertheless, for some important problems regarding analysis of facial behaviour, such as (a) estimation of affect in a continuous dimensional space (e.g., valence and arousal) in videos displaying spontaneous facial behaviour and (b) detection of the activated facial muscles (i.e., facial action unit detection), to the best of our knowledge, well-established in-the-wild databases and benchmarks do not exist. That is, the majority of the publicly available corpora for the above tasks contain samples that have been captured in controlled recording conditions and/or captured under a very specific milieu. Arguably, in order to make further progress in automatic understanding of facial behaviour, datasets that have been captured in in-the-wild and in various milieus have to be developed. In this paper, we survey the progress that has been recently made on understanding facial behaviour inthe-wild, namely the datasets and methodologies that have been developed thus far, while paying particular attention to recently proposed deep learning techniques. Finally, we attempt a significant step further by proposing a novel, comprehensive benchmark that can be utilized for evaluating and training various methodologies for the problems of facial affect, behaviour analysis and understanding "inthe-wild". To the best of our knowledge, this is the first benchmark proposed for measuring continuous affect in the valence-arousal space "in-the-wild".

Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark

“…Early datasets for FER such as JAFFE [44], CK [42,58], MMI [50], and MultiPie [2] were captured in a lab-controlled environment. Thus, the datasets collected in the wild condition, which contains the people's naturalistic emotion states [4,8,19,20,33,45,47,48,64] have attracted much more attention. Specifically, the AFEW [19] includes video clips extracted from movies and SFEW [20] is built using static images from the subset video clips of AFEW.…”

Learners’ Efficiency Prediction Using Facial Behavior Analysis

“…Many works have been carried out for FBA using different facial behavior features and have applied it to various applications [17,18,19]. Bhatia et al used FBA in order to distinguish melancholia and non-melancholia subjects as it mainly depends on facial affect and mood [20].…”

User Adaptive and Context-Aware Smart Home Using Pervasive and Semantic Technologies

“…The contributions of the already developed datasets and benchmarks for analysis of facial expression in the wild have been demonstrated during the challenges in Representation Learning (ICML 2013) [67], in the series of Emotion Recognition in the wild challenges (EmotiW 2013, 2014, 2015 [61,[68][69][70], and 2016 (https://sites.google.com/ site/emotiw2016/)) and in the recently organized workshop on context-based affect recognition (CBAR 2016 (http:// cbar2016.blogspot.gr/)). For a more extended overview on datasets collected in the wild, the reader is referred to [71].…”