Emotion recognition from audio-visual data using rule based decision level fusion

Sahoo, Subhasmita; Routray, Aurobinda

doi:10.1109/techsym.2016.7872646

Cited by 29 publications

(15 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Data fusion is a critical step involved in multimodal emotion recognition for producing the estimation. The literature about emotional data fusion involves three data fusion techniques, which are early fusion (feature fusion) [42,43], late fusion (decision fusion) [44,45,46] and hybrid approaches [17,47,48].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

“…These outputs can be used separately or jointly through machine learning methods. Some studies have implemented decision level fusion as reported in [44,45,49,54,55].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

“…The study in [44] a decision level fusion is applied for audio and visual data to identify emotions. The proposed method is applied on eNTERFACE'05 database [56].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

See 2 more Smart Citations

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

show abstract

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

“…These outputs can be used separately or jointly through machine learning methods. Some studies have implemented decision level fusion as reported in [44,45,49,54,55].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

show abstract

“…The basic LBP function takes into account the eight neighborhoods of a pixel, whereas the description includes all circular neighborhoods [38]. The notation (P, R) in Fig.1, represents the pixel neighborhoods P, sampling points on a circle of radius R [38], [39]. P-bit binary number shown in Fig.…”

Section: A One Dimensional Local Binary Patternmentioning

confidence: 99%

A Speech Emotion Recognition Model Based on Multi-Level Local Binary and Local Ternary Patterns

Sönmez

Varol

2020

IEEE Access

View full text Add to dashboard Cite

Interpreting a speech signal is quite challenging because it consists of different frequencies and features that vary according to emotions. Although different algorithms are being developed in the speech emotion recognition (SER) domain, the success rates vary according to the spoken languages, emotions, and databases. In this study, a new lightweight effective SER method has been developed that has low computational complexity. This method, called 1BTPDN, is applied on RAVDESS, EMO-DB, SAVEE, and EMOVO databases. First, low-pass filter coefficients are obtained by applying a one-dimensional discrete wavelet transform on the raw audio data. The features are extracted by applying textural analysis methods, a one-dimensional local binary pattern, and a one-dimensional local ternary pattern to each filter. Using neighborhood component analysis, the most dominant 1024 features are selected from 7680 features while the other features are discarded. These 1024 features are selected as the input of the classifier which is a thirddegree polynomial kernel-based support vector machine. The success rates of the 1BTPDN reached 95.16%, 89.16%, 76.67%, and 74.31% in the RAVDESS, EMO-DB, SAVEE, and EMOVO databases, respectively. The recognition rates are higher compared to many textural, acoustic, and deep learning state-of-the-art SER methods. INDEX TERMS Discrete wavelet transform, local binary pattern, local ternary pattern, neighborhood component analysis, speech emotion recognition.

show abstract

“…Recently, studies [42,43] have examined rule-based emotion recognition from audiovisual data in the context of fusing outcomes from individual modalities (rule-based decision-level emotion recognition or late binding). However, the rules were not used to extract features in these studies.…”

Section: Introductionmentioning

confidence: 99%

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Patwardhan¹

2017

MTI

View full text Add to dashboard Cite

This paper presents a multimodal emotion recognition method that uses a feature-level combination of three-dimensional (3D) geometric features (coordinates, distance and angle of joints), kinematic features such as velocity and displacement of joints, and features extracted from daily behavioral patterns such as frequency of head nod, hand wave, and body gestures that represent specific emotions. Head, face, hand, body, and speech data were captured from 15 participants using an infrared sensor (Microsoft Kinect). The 3D geometric and kinematic features were developed using raw feature data from the visual channel. Human emotional behavior-based features were developed using inter-annotator agreement and commonly observed expressions, movements and postures associated to specific emotions. The features from each modality and the behavioral pattern-based features (head shake, arm retraction, body forward movement depicting anger) were combined to train the multimodal classifier for the emotion recognition system. The classifier was trained using 10-fold cross validation and support vector machine (SVM) to predict six basic emotions. The results showed improvement in emotion recognition accuracy (The precision increased by 3.28% and the recall rate by 3.17%) when the 3D geometric, kinematic, and human behavioral pattern-based features were combined for multimodal emotion recognition using supervised classification.

show abstract

Emotion recognition from audio-visual data using rule based decision level fusion

Cited by 29 publications

References 15 publications

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

A Speech Emotion Recognition Model Based on Multi-Level Local Binary and Local Ternary Patterns

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Contact Info

Product

Resources

About