Audio-visual emotion fusion (AVEF): A deep efficient weighted approach

Ma, Yaxiong; Hao, Yixue; Chen, Min; Chen, Jincai; Lü, Ping; Košir, Andrej

doi:10.1016/j.inffus.2018.06.003

Cited by 127 publications

(54 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, transfer learning methods play a very limited role in this process. Common knowledge transfer in multi-modal methods include fine-tune well trained models to a specific type of signal (Vielzeuf et al, 2017;Yan et al, 2018;Huang et al, 2019;Ortega et al, 2019), or fine-tune different well-trained models to both speech and video signals (Ouyang et al, 2017;Zhang et al, 2017;Ma et al, 2019). Other usage of transfer learning for multi-modal methods includes leveraging the knowledge from one signal to another (e.g., video to speech) to reduce the potential bias (Athanasiadis et al, 2019).…”

Section: Multi-modal Transfer Learning For Emotion Recognitionmentioning

confidence: 99%

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Feng

Chaspari

2020

Front. Comput. Sci.

View full text Add to dashboard Cite

Automatic emotion recognition is the process of identifying human emotion from signals such as facial expression, speech, and text. Collecting and labeling such signals is often tedious and many times requires expert knowledge. An effective way to address challenges related to the scarcity of data and lack of human labels, is transfer learning. In this manuscript, we will describe fundamental concepts in the field of transfer learning and review work which has successfully applied transfer learning for automatic emotion recognition. We will finally discuss promising future research directions of transfer learning for improving the generalizability of automatic emotion recognition systems.

show abstract

Section: Multi-modal Transfer Learning For Emotion Recognitionmentioning

confidence: 99%

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Feng

Chaspari

2020

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…However, bimodal emotion recognition can reach an accuracy of 86.85%, an increase of 5% compared with using a single modal of emotion recognition (Song et al 2015;Chuang and Wu 2004;Kessous et al 2010). Furthermore, previous studies have indicated that it is impossible to achieve satisfactory results by recognizing emotions based on a single model for either speech or facial expression (Ma et al 2019;Yang et al 2017). Accordingly, this study applied a bimodal emotion recognition system by using both facial expression recognition (55%) and speech recognition (45%).…”

Section: Methodsmentioning

confidence: 99%

Development and evaluation of an emotional lexicon system for young children

Wei

2019

Microsyst Technol

View full text Add to dashboard Cite

Traditionally, children's emotion has been assessed by teachers according to observation. We should be able to detect children's emotions using algorithmic techniques. To achieve this goal, it is necessary to develop and evaluate an emotional lexicon based on the standardized test entitled Emotional Competencies Scale for Young Children (ECSYC). The purpose of this study was to establish the criterion-related validity. The methodology of this study was to firstly develop 40 scenarios based on ECSYC. Secondly, we developed the five-level criteria. Thirdly, this study implemented observer training and calculated inter-rater consistency reliability. Fourthly, observers categorized 200 children's replies into five levels. Fifthly, this study ranked the sequence of frequency of each level and completed the emotional lexicon. The findings showed that the Spearman's rho coefficient reached up to .406*. (p = .026), which is significant, indicating that Young Children Emotional Lexicon (YCEL) and ECSYC were significantly correlated. The accuracies of the emotion detection recognizer using a bimodal emotion recognition approach achieved 46.7%, 60.85% and 78.73% for facial expression recognition, speech recognition, and a bimodal emotion recognition, respectively. Findings confirmed that the YCEL is feasible for speech recognition. The bimodal emotion recognition accuracies increased 32.03% and 17.88% compared with using a single modal of facial expression recognition and speech recognition, respectively. It is feasible to automatically detect children's emotional development and bring the norm up to date.

show abstract

“…Recently, a little work has investigated multiple modalities to recognize emotions [9][10][11][12]. Many studies have fused facial expression together with physiological signals [30]. Table 3 [30][31][32][33] presents some of the surveyed studies on the multi-modal emotion fusion, including the corpus, modalities of fusion, feature extractors, fusion approach, classifier, and classification accuracy.…”

Section: Literature Review On Multi-modal Emotion Recognitionmentioning

confidence: 99%

“…Many studies have fused facial expression together with physiological signals [30]. Table 3 [30][31][32][33] presents some of the surveyed studies on the multi-modal emotion fusion, including the corpus, modalities of fusion, feature extractors, fusion approach, classifier, and classification accuracy. As indicated in the literature and to the best of our knowledge, there are no works reported on fusing speech with EEG for recognizing emotions.…”

Section: Literature Review On Multi-modal Emotion Recognitionmentioning

confidence: 99%

Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information

2019

View full text Add to dashboard Cite

In multi-modal emotion aware frameworks, it is essential to estimate the emotional features then fuse them to different degrees. This basically follows either a feature-level or decision-level strategy. In all likelihood, while features from several modalities may enhance the classification performance, they might exhibit high dimensionality and make the learning process complex for the most used machine learning algorithms. To overcome issues of feature extraction and multi-modal fusion, hybrid fuzzy-evolutionary computation methodologies are employed to demonstrate ultra-strong capability of learning features and dimensionality reduction. This paper proposes a novel multi-modal emotion aware system by fusing speech with EEG modalities. Firstly, a mixing feature set of speaker-dependent and independent characteristics is estimated from speech signal. Further, EEG is utilized as inner channel complementing speech for more authoritative recognition, by extracting multiple features belonging to time, frequency, and time–frequency. For classifying unimodal data of either speech or EEG, a hybrid fuzzy c-means-genetic algorithm-neural network model is proposed, where its fitness function finds the optimal fuzzy cluster number reducing the classification error. To fuse speech with EEG information, a separate classifier is used for each modality, then output is computed by integrating their posterior probabilities. Results show the superiority of the proposed model, where the overall performance in terms of accuracy average rates is 98.06%, and 97.28%, and 98.53% for EEG, speech, and multi-modal recognition, respectively. The proposed model is also applied to two public databases for speech and EEG, namely: SAVEE and MAHNOB, which achieve accuracies of 98.21% and 98.26%, respectively.

show abstract

Audio-visual emotion fusion (AVEF): A deep efficient weighted approach

Cited by 127 publications

References 20 publications

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Development and evaluation of an emotional lexicon system for young children

Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information

Contact Info

Product

Resources

About