Facial Expression and Gesture Analysis for Emotionally-Rich Man-Machine Interaction

Karpouzis, Kostas; Raouzaiou, Amaryllis; Drosopoulos, Athanasios; Ioannou, Spiros; Balomenos, Themis; Tsapatsoulis, Nicolas; Kollias, Stefanos

doi:10.4018/978-1-59904-941-0.ch128

Cited by 10 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Karpouzis et al [58] [9] used data labelled using a continuous dimensional space (valence and activity) and mapped it onto 3 classes corresponding to 3 of the 4 quadrants in the valence-arousal plane (only 3 quadrants were necessary since little data was included in the left quadrant). This resulted in a 3-classes problem.…”

Section: Discussionmentioning

confidence: 99%

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Kessous¹,

Castellano

2009

J Multimodal User Interfaces

212

View full text Add to dashboard Cite

In this paper a study on multimodal automatic emotion recognition during a speech-based interaction is presented. A database was constructed consisting of people pronouncing a sentence in a scenario where they interacted with an agent using speech. Ten people pronounced a sentence corresponding to a command while making 8 different emotional expressions. Gender was equally represented, with speakers of several different native languages including French, German, Greek and Italian. Facial expression, gesture and acoustic analysis of speech were used to extract features relevant to emotion. For the automatic classification of unimodal data, bimodal data and multimodal data, a system based on a Bayesian classifier was used. After performing an automatic classification of each modality, the different modalities were combined using a multimodal approach. Fusion of the modalities at the feature level (before running the classifier) and at the results level (combining results from classifier from each modality) were compared. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison to the unimodal systems: the multimodal approach increased the recognition rate by more than 10% when compared to the most successful unimodal system. Bimodal emotion recognition based on all combinations of the modalities (i.e., 'face-gesture', 'facespeech' and 'gesture-speech') was also investigated. The results show that the best pairing is 'gesture-speech'. Using all three modalities resulted in a 3.3% classification improvement over the best bimodal results.

show abstract

Section: Discussionmentioning

confidence: 99%

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Kessous¹,

Castellano

2009

J Multimodal User Interfaces

212

View full text Add to dashboard Cite

show abstract

“…In order to extract emotion-related features through hand movement, we implemented a lightweight hand-tracking subsystem [22,51]. The purpose of this subsystem was not gesture recognition per se, but the extraction of quantitative parameters related to expressivity [12,7]; as a result, emphasis is on detailed tracking, e.g.…”

Section: Hand Gesture Analysismentioning

confidence: 99%

“…Facial expressions [21], [50], [31], vocal features [41], [24] and physiological signals [48] have been used as inputs during these attempts, while multimodal emotion recognition is currently gaining ground (26,28,32,53).…”

Section: Introductionmentioning

confidence: 99%

Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition

Karpouzis

Caridakis

Kessous³

et al.

Artifical Intelligence for Human Computing

Self Cite

View full text Add to dashboard Cite

Abstract. Affective and human-centered computing have attracted a lot of attention during the past years, mainly due to the abundance of devices and environments able to exploit multimodal input from the part of the users and adapt their functionality to their preferences or individual habits. In the quest to receive feedback from the users in an unobtrusive manner, the combination of facial and hand gestures with prosody information allows us to infer the users' emotional state, relying on the best performing modality in cases where one modality suffers from noise or bad sensing conditions. In this paper, we describe a multi-cue, dynamic approach to detect emotion in naturalistic video sequences. Contrary to strictly controlled recording conditions of audiovisual material, the proposed approach focuses on sequences taken from nearly real world situations. Recognition is performed via a 'Simple Recurrent Network' which lends itself well to modeling dynamic events in both user's facial expressions and speech. Moreover this approach differs from existing work in that it models user expressivity using a dimensional representation of activation and valence, instead of detecting discrete 'universal emotions', which are scarce in everyday human-machine interaction. The algorithm is deployed on an audiovisual database which was recorded simulating human-human discourse and, therefore, contains less extreme expressivity and subtle variations of a number of emotion labels.

show abstract

“…More information about the expression profile based fuzzy recognition system can be found in [15], [16].…”

Section: B Mask Extractionmentioning

confidence: 99%

Intelligent Facial Analysis and Expression Recognition

Ioannou¹,

Wallace²,

Kollias³

The 2006 IEEE International Joint Conference on Neural Network Proceedings

View full text Add to dashboard Cite

Abstract-Since facial expressions are a key modality in human communication, the automated analysis of facial images and video for the estimation of the displayed expression is central in the design of intuitive and human friendly computer interaction systems. In this paper we present an intelligent feature extraction system which combines analysis from multiple channels based on their confidence, to result in better, error resilient facial feature boundary detection. Neural networks are a key component of the system. Issues such as uncertainty and lack of confidence in the process of feature extraction are considered during the expression analysis and recognition. Various results are presented which illustrate the performance of the method.

show abstract

Facial Expression and Gesture Analysis for Emotionally-Rich Man-Machine Interaction

Cited by 10 publications

References 0 publications

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition

Intelligent Facial Analysis and Expression Recognition

Contact Info

Product

Resources

About