Audio-Visual Affect Recognition in Activation-Evaluation Space

Abstract

The ability of a computer to detect and appropriately respond to changes in a user's affective state has significant implications to Human-Computer Interaction (HCI). To more accurately simulate the human ability to assess affects through multi-sensory data, automatic affect recognition should also make use of multimodal data. In this paper, we present our efforts toward audio-visual affect recognition. Based on psychological research, we have chosen affect categories based on an activationevaluation space which is robust in capturing significant aspects of emotion. We apply the Fisher boosting learning algorithm which can build a strong classifier by combining a small set of weak classification functions. Our experimental results show with 30 Fisher features, the testing error rates of our bimodal affect recognition is about 16% on the evaluation axis and 13% on the activation axis.

Speech-based recognition of self-reported and observed emotion in a dimensional space

“…Yu et al (2004) classified user engagement in social telephone conversations between friends along arousal and valence scales that were discretized into 5 levels. Kim et al (2005), Zeng et al (2005) andWo ¨llmer et al (2009) classified emotions in the 4 emotion quadrants of the arousal-valence space. Instead of classifying emotions on discretized scales of arousal and valence, some studies have taken up the challenge to classify emotions on continuous scales of arousal and valence.…”

Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey

“…• Audio-Visual Classification may be useful in identifying affective cues such as vocal inflections or prosodic features [180], [188], [223].…”

“…Among the audiovisual affect recognition approaches, Busso et al [190], Petridis and Pantic [184] and Schuller et al [183] have employed feature-level fusion, which concatenates multimodal features and passes them through a single affect recognition model or classifier. Decision-level fusion, on the other hand, has been used by Hoch et al [255], Go et al [187], Pal et al [185], Wang and Guan [180], Zeng et al [178], Zeng et al [256], and Zeng et al [223]. However, decision-level fusion ignores the inherent correlation between these multi-modal features.…”

“…Wang & Guan [180] Gabor wavelets Fisher's-LDA Prosody, MFCC Zeng et al [179] Motion units, Prosody MFHMM Zeng et al [178] LLP, Prosody Adaboost + MHMM Zeng et al [256] Motion units, Prosody SNoW Zeng et al [221] Motion units, Prosody MFHMM Zeng et al [223] Motion units, Prosody HMM Wöllmeret al [192] MFCC, Facial flow LSTM Karpouzis et al [186] FPs, Prosody RNN Caridakis et al [189] Facial points, Prosody RNN Fragopanagos & FAPs, Prosody Neural network Taylor [188] (NN) Petridis & Facial points, MFCC Adaboost, NN Pantic [184] Ringeval et al [191] Low level descriptor -MFCC , LGBP from three orthogonal planes Schoneveld et al [219] Face Image, MFCC CNN->LSTM Hossain &…”

“…MFCC & Image CNN->ELM->SVM Muhammad [251] TABLE 9: Audio-visual analysis for affect recognition is categorized based on features and techniques. [190], [255] and different types of HMMs [178], [179], [181], [221], [223].…”