Learning a sparse codebook of facial and body microexpressions for emotion recognition

Abstract

Obtaining a compact and discriminative representation of facial and body expressions is a difficult problem in emotion recognition. Part of the difficulty is capturing microexpressions, i.e., short, involuntary expressions that last for only a fraction of a second: at a micro-temporal scale, there are so many other subtle face and body movements that do not convey semantically meaningful information. We present a novel approach to this problem by exploiting the sparsity of the frequent micro-temporal motion patterns. Local space-time features are extracted over the face and body region for a very short time period, e.g., few milliseconds. A codebook of microexpressions is learned from the data and used to encode the features in a sparse manner. This allows us to obtain a representation that captures the most salient motion patterns of the face and body at a micro-temporal scale. Experiments performed on the AVEC 2012 dataset show our approach achieving the best published performance on the expectation dimension based solely on visual features. We also report experimental results on audio-visual emotion recognition, comparing early and late data fusion techniques.

Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection

“…Using the non-parametric Wilcoxon test, we found that the difference between correlations resulted from LSTM-RNN and CCRF is not significant, however, RMSE is significantly lower for LSTM-RNN (p < 1E − 4). Although, direct comparison of the performance is not possible with the other work due to the difference in the nature of the databases, the best achieved correlation is in the same range as the result of [46], the winner of AVEC 2012 challenge, on valence and superior to the correlation value reported on valence in a more recent work, [47]. Unfortunately, the previous papers on this topic did not report the standard deviation of their results; thus its comparison was impossible.…”

Continuous emotion detection using EEG signals and facial expressions

“…Therefore, we conclude that the LSTM-RNN performed the best in this setting and with the goal of continuous valence detection. Although direct comparison of the performance is not possible with the other works due to the difference in the nature of the databases, the best achieved correlation is in the same range as the result of [23], the winner of AVEC 2012 challenge, on valence and superior to the correlation value reported on valence in a more recent work, [24]. Unfortunately, the previous papers on this topic did not report the standard deviation of their results; thus its comparison was impossible.…”

Real Time Multimodal Emotion Recognition System using Facial Landmarks and Hand over Face Gestures

“…Recently, Song et al [12] used them to encode facial and body micro expressions for emotion detection. They reflect interesting events that can be used for a compact representation of video data as well as for its interpretation.…”

“…They reflect interesting events that can be used for a compact representation of video data as well as for its interpretation. We used the approach proposed by Song et al [12]. STIP capture salient visual patterns in a space-time image volume by extending the local spatial image descriptor to the space-time domain.…”