A Multimodal Emotion Recognition System Using Facial Landmark Analysis

Rahdari, Farhad; Rashedi, Esmat; Eftekhari, Mohammad

doi:10.1007/s40998-018-0142-9

Cited by 23 publications

(9 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recognition Methods. To demonstrate the performance of the proposed method, the results of this paper are compared with those of References [16,19,20], as shown in Table 2.…”

Section: Comparison Of Results From Multiple Emotionmentioning

confidence: 99%

“…However, the proposed method cannot quickly locate the emotional key information from the multimodal signals containing a large amount of redundant information, so the overall efficiency of the model needs to be improved. In Reference [20], a novel visual expression feature extraction and classification method based on the displacement of specific continuous landmarks in the continuous frame of voice is proposed.…”

Section: Related Workmentioning

confidence: 99%

“…Significant Features. In the learning phase of facial expression features, four methods (proposed method and References [14,20,28]) were used to compare the results of feature learning based on the DEAP dataset in the two dimensions of arousal and valence, respectively, as shown in Figures 8 and 9. The R-squared coefficient and loss function Loss are used as the evaluation indicators of feature learning.…”

Section: Comparison Of the Learning Effect On Facial Emotionmentioning

confidence: 99%

“…Results of Other Methods. The two-dimensional emotional spatial expression was used to compare the results of dimensional affective recognition with those of References [16,19,20] in the MAHNOB-HCI dataset, as shown in Table 1.…”

Section: Compared With the Dimensional Emotion Recognitionmentioning

confidence: 99%

See 3 more Smart Citations

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Zhao

Chen

2021

Computational and Mathematical Methods in Medicine

View full text Add to dashboard Cite

Due to the complexity of human emotions, there are some similarities between different emotion features. The existing emotion recognition method has the problems of difficulty of character extraction and low accuracy, so the bidirectional LSTM and attention mechanism based on the expression EEG multimodal emotion recognition method are proposed. Firstly, facial expression features are extracted based on the bilinear convolution network (BCN), and EEG signals are transformed into three groups of frequency band image sequences, and BCN is used to fuse the image features to obtain the multimodal emotion features of expression EEG. Then, through the LSTM with the attention mechanism, important data is extracted in the process of timing modeling, which effectively avoids the randomness or blindness of sampling methods. Finally, a feature fusion network with a three-layer bidirectional LSTM structure is designed to fuse the expression and EEG features, which is helpful to improve the accuracy of emotion recognition. On the MAHNOB-HCI and DEAP datasets, the proposed method is tested based on the MATLAB simulation platform. Experimental results show that the attention mechanism can enhance the visual effect of the image, and compared with other methods, the proposed method can extract emotion features from expressions and EEG signals more effectively, and the accuracy of emotion recognition is higher.

show abstract

“…Recognition Methods. To demonstrate the performance of the proposed method, the results of this paper are compared with those of References [16,19,20], as shown in Table 2.…”

Section: Comparison Of Results From Multiple Emotionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Comparison Of the Learning Effect On Facial Emotionmentioning

confidence: 99%

Section: Compared With the Dimensional Emotion Recognitionmentioning

confidence: 99%

See 2 more Smart Citations

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Zhao

Chen

2021

Computational and Mathematical Methods in Medicine

View full text Add to dashboard Cite

show abstract

“…In this regard, several multi-modal emotion recognition approaches have achieved significant contributions [4][5][6]. Though there are distinctive input modalities for recognizing emotions, the most widely recognized are bimodal inputs that combine both speech and video.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information

2019

View full text Add to dashboard Cite

In multi-modal emotion aware frameworks, it is essential to estimate the emotional features then fuse them to different degrees. This basically follows either a feature-level or decision-level strategy. In all likelihood, while features from several modalities may enhance the classification performance, they might exhibit high dimensionality and make the learning process complex for the most used machine learning algorithms. To overcome issues of feature extraction and multi-modal fusion, hybrid fuzzy-evolutionary computation methodologies are employed to demonstrate ultra-strong capability of learning features and dimensionality reduction. This paper proposes a novel multi-modal emotion aware system by fusing speech with EEG modalities. Firstly, a mixing feature set of speaker-dependent and independent characteristics is estimated from speech signal. Further, EEG is utilized as inner channel complementing speech for more authoritative recognition, by extracting multiple features belonging to time, frequency, and time–frequency. For classifying unimodal data of either speech or EEG, a hybrid fuzzy c-means-genetic algorithm-neural network model is proposed, where its fitness function finds the optimal fuzzy cluster number reducing the classification error. To fuse speech with EEG information, a separate classifier is used for each modality, then output is computed by integrating their posterior probabilities. Results show the superiority of the proposed model, where the overall performance in terms of accuracy average rates is 98.06%, and 97.28%, and 98.53% for EEG, speech, and multi-modal recognition, respectively. The proposed model is also applied to two public databases for speech and EEG, namely: SAVEE and MAHNOB, which achieve accuracies of 98.21% and 98.26%, respectively.

show abstract

Social Media Multi-modal Processing Mode for Emergency

Lin

2022

Application of Intelligent Systems in Multi-Modal Information Analytics

View full text Add to dashboard Cite

A Multimodal Emotion Recognition System Using Facial Landmark Analysis

Cited by 23 publications

References 44 publications

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information

Social Media Multi-modal Processing Mode for Emergency

Contact Info

Product

Resources

About