Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering

Savran, Arman; Cao, Huiluo; Shah, Miraj; Nenkova, Ani; Verma, Ragini

doi:10.1145/2388676.2388781

Cited by 52 publications

(36 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We showed that LZM and QLZM representations are useful, but the most appropriate representation for affect recognition in posed and naturalistic settings may differ (unlike the common practice: [33] vs. [30,32]). …”

Section: Resultsmentioning

confidence: 94%

Local Zernike Moment Representation for Facial Affect Recognition

Sarıyanidi¹,

Güneş²,

Gokmen³

et al. 2013

Procedings of the British Machine Vision Conference 2013

View full text Add to dashboard Cite

In this paper, we propose to use local Zernike Moments (ZMs) for facial affect recognition and introduce a representation scheme based on performing non-linear encoding on ZMs via quantization. Local ZMs provide a useful and compact description of image discontinuities and texture. We demonstrate the use of this ZM-based representation for posed and discrete as well as naturalistic and continuous affect recognition on standard datasets, and show that ZM-based representations outperform well-established alternative approaches for both tasks. To the best of our knowledge, the performance we achieved on CK+ dataset is superior to all results reported to date.

show abstract

Section: Resultsmentioning

confidence: 94%

Local Zernike Moment Representation for Facial Affect Recognition

Sarıyanidi¹,

Güneş²,

Gokmen³

et al. 2013

Procedings of the British Machine Vision Conference 2013

View full text Add to dashboard Cite

show abstract

“…Table 1 shows the cross-correlation coefficients between predicted and ground-truth labels, averaged over all sequences. We include the baseline result from [37] and the results of the top four contenders from AVEC 2012 [27,41,36,30]. We also include our results on audio-visual input (bottom row), discussed in Section 4.4.…”

Section: Methodsmentioning

confidence: 99%

Learning a sparse codebook of facial and body microexpressions for emotion recognition

Song¹,

Morency²,

Davis³

2013

Proceedings of the 15th ACM on International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Obtaining a compact and discriminative representation of facial and body expressions is a difficult problem in emotion recognition. Part of the difficulty is capturing microexpressions, i.e., short, involuntary expressions that last for only a fraction of a second: at a micro-temporal scale, there are so many other subtle face and body movements that do not convey semantically meaningful information. We present a novel approach to this problem by exploiting the sparsity of the frequent micro-temporal motion patterns. Local space-time features are extracted over the face and body region for a very short time period, e.g., few milliseconds. A codebook of microexpressions is learned from the data and used to encode the features in a sparse manner. This allows us to obtain a representation that captures the most salient motion patterns of the face and body at a micro-temporal scale. Experiments performed on the AVEC 2012 dataset show our approach achieving the best published performance on the expectation dimension based solely on visual features. We also report experimental results on audio-visual emotion recognition, comparing early and late data fusion techniques.

show abstract

“…al. [20] use temporal statistics of texture descriptors extracted from facial videos, a combination of various acoustic features, and lexical features to create regression based affect estimators for each modality. The single modality regressors are then combined using particle filtering, by treating these independent regression outputs as measurements of the affective states in a Bayesian filtering framework, where previous observations provide prediction about the current state by means of learned affect dynamics.…”

Section: Related Workmentioning

confidence: 99%

Automatic affective dimension recognition from naturalistic facial expressions based on wavelet filtering and PLS regression

Gaus

Meng

Jan

et al. 2015

2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)

View full text Add to dashboard Cite

Abstract-Automatic affective dimension recognition from facial expression continuously in naturalistic contexts is a very challenging research topic but very important in humancomputer interaction. In this paper, an automatic recognition system was proposed to predict the affective dimensions such as Arousal, Valence and Dominance continuously in naturalistic facial expression videos. Firstly, visual and vocal features are extracted from image frames and audio segments in facial expression videos. Secondly, a wavelet transform based digital filtering method is applied to remove the irrelevant noise information in the feature space. Thirdly, Partial Least Squares regression is used to predict the affective dimensions from both video and audio modalities. Finally, two modalities are combined to boost overall performance in the decision fusion process. The proposed method is tested in the fourth international Audio/Visual Emotion Recognition Challenge (AVEC2014) dataset and compared to other state-of-the-art methods in the affect recognition sub-challenge with a good performance.

show abstract

Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering

Cited by 52 publications

References 19 publications

Local Zernike Moment Representation for Facial Affect Recognition

Local Zernike Moment Representation for Facial Affect Recognition

Learning a sparse codebook of facial and body microexpressions for emotion recognition

Automatic affective dimension recognition from naturalistic facial expressions based on wavelet filtering and PLS regression

Contact Info

Product

Resources

About