MSER: Multimodal speech emotion recognition using cross-attention with deep fusion

Khan, Mustaqeem; Gueaieb, Wail; El Saddik, Abdulmotaleb; Kwon, Soonil

doi:10.1016/j.eswa.2023.122946

Cited by 22 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The widely used "scaled dot-product attention" [53] practically requires transforming the three matrices of queries (Qs), keys (Ks), and values (Vs) into an output vector. In general, it computes the dot product of the query Q with all keys Ks, which is then divided by √ d k (d k , the dimension of queries), and finally, a softmax layer ( f softmax ) is applied to tune the weights on the values V. The attention can be computed as shown in Equation (15).…”

Section: Cross-attention Fusion Of Encoded Cuesmentioning

confidence: 99%

“…Liu et al incorporate EEG, peripheral physiological signals, electrocardiogram, and movement features, and deep canonical correlation analysis and bimodal auto-encoder are extended for multi-modal signal fusion [13]. Emotion recognition has also been devoted to fusing acoustic, linguistic, and textual information [14], such as speech-text dual-modal frameworks [15,16], audio-visual-text tri-modal frameworks [17], and multi-modal frameworks using audio, text, facial expressions, and hand movements [18].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

Yu,

Meng,

Fan

et al. 2024

Electronics

View full text Add to dashboard Cite

Speech emotion recognition (SER) aims to recognize human emotions through in-depth analysis of audio signals. However, it remains challenging to encode emotional cues and to fuse the encoded cues effectively. In this study, dual-stream representation is developed, and both full training and fine-tuning of different deep networks are employed for encoding emotion patterns. Specifically, a cross-attention fusion (CAF) module is designed to integrate the dual-stream output for emotion recognition. Using different dual-stream encoders (fully training a text processing network and fine-tuning a pre-trained large language network), the CAF module is compared to other three fusion modules on three databases. The SER performance is quantified with weighted accuracy (WA), unweighted accuracy (UA), and F1-score (F1S). The experimental results suggest that the CAF outperforms the other three modules and leads to promising performance on the databases (EmoDB: WA, 97.20%; UA, 97.21%; F1S, 0.8804; IEMOCAP: WA, 69.65%; UA, 70.88%; F1S, 0.7084; RAVDESS: WA, 81.86%; UA, 82.75.21%; F1S, 0.8284). It is also found that fine-tuning a pre-trained large language network achieves superior representation than fully training a text processing network. In a future study, improved SER performance could be achieved through the development of a multi-stream representation of emotional cues and the incorporation of a multi-branch fusion mechanism for emotion recognition.

show abstract

Section: Cross-attention Fusion Of Encoded Cuesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

Yu,

Meng,

Fan

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…However, this method will inevitably result in huge computational and memory costs. In recent years, deep learning methodologies, particularly those involving deep convolutional neural networks (CNNs), have demonstrated exceptional prowess across various domains, including image restoration [58] and speech signal processing [59]. The fusion of deep learning techniques with sparse SAR imaging will emerge as a promising avenue for future research.…”

Section: Introductionmentioning

confidence: 99%

A Sparse SAR Imaging Method for Low-Oversampled Staggered Mode via Compound Regularization

Liu,

Pan,

Zhu

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

High-resolution wide-swath (HRWS) imaging is the research focus of the modern spaceborne synthetic-aperture radar (SAR) imaging field, with significant relevance and vast application potential. Staggered SAR, as an innovative imaging system, mitigates blind areas across the entire swath by periodically altering the radar pulse repetition interval (PRI), thereby extending the swath width to multiples of that achievable by conventional systems. However, the staggered mode introduces inherent challenges, such as nonuniform azimuth sampling and echo data loss, leading to azimuth ambiguities and substantially impacting image quality. This paper proposes a sparse SAR imaging method for the low-oversampled staggered mode via compound regularization. The proposed method not only effectively suppresses azimuth ambiguities arising from nonuniform sampling without necessitating the restoration of missing echo data, but also incorporates total variation (TV) regularization into the sparse reconstruction model. This enhances the accurate reconstruction of distributed targets within the scene. The efficacy of the proposed method is substantiated through simulations and real data experiments from spaceborne missions.

show abstract

A Novel Dual Kernel Support Vector-Based Levy Dung Beetle Algorithm for Accurate Speech Emotion Detection

Han,

Zhang,

Ren

et al. 2024

Circuits Syst Signal Process

View full text Add to dashboard Cite

MSER: Multimodal speech emotion recognition using cross-attention with deep fusion

Cited by 22 publications

References 22 publications

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

A Sparse SAR Imaging Method for Low-Oversampled Staggered Mode via Compound Regularization

A Novel Dual Kernel Support Vector-Based Levy Dung Beetle Algorithm for Accurate Speech Emotion Detection

Contact Info

Product

Resources

About