Advanced Fusion-Based Speech Emotion Recognition System Using a Dual-Attention Mechanism with Conv-Caps and Bi-GRU Features

Abstract

Recognizing the speaker’s emotional state from speech signals plays a very crucial role in human–computer interaction (HCI). Nowadays, numerous linguistic resources are available, but most of them contain samples of a discrete length. In this article, we address the leading challenge in Speech Emotion Recognition (SER), which is how to extract the essential emotional features from utterances of a variable length. To obtain better emotional information from the speech signals and increase the diversity of the information, we present an advanced fusion-based dual-channel self-attention mechanism using convolutional capsule (Conv-Cap) and bi-directional gated recurrent unit (Bi-GRU) networks. We extracted six spectral features (Mel-spectrograms, Mel-frequency cepstral coefficients, chromagrams, the contrast, the zero-crossing rate, and the root mean square). The Conv-Cap module was used to obtain Mel-spectrograms, while the Bi-GRU was used to obtain the rest of the spectral features from the input tensor. The self-attention layer was employed in each module to selectively focus on optimal cues and determine the attention weight to yield high-level features. Finally, we utilized a confidence-based fusion method to fuse all high-level features and pass them through the fully connected layers to classify the emotional states. The proposed model was evaluated on the Berlin (EMO-DB), Interactive Emotional Dyadic Motion Capture (IEMOCAP), and Odia (SITB-OSED) datasets to improve the recognition rate. During experiments, we found that our proposed model achieved high weighted accuracy (WA) and unweighted accuracy (UA) values, i.e., 90.31% and 87.61%, 76.84% and 70.34%, and 87.52% and 86.19%, respectively, demonstrating that the proposed model outperformed the state-of-the-art models using the same datasets.

Recognition of Emotion with Intensity from Speech Signal Using 3D Transformed Feature and Deep Learning

“…Latif et al [21] also employed several DL models on raw speech. Several modified DL models are also observed for SER, including capsule neural network [22], 3D CNN-LSTM model [10], 3D CNN using k-means clustering [8], 2D CNN with a self-attention dilated residual network [9], Spiking Neural Network (SNN) [23], convolutional capsule (Conv-Cap) and bi-directional gated recurrent unit (Bi-GRU) [24], attention-LSTM-attention [25], Bi-GRU with attention mechanism [26], CNN with a capsule neural network (Caps Net) [13], temporal CNN with self-attention transfer network (SATN) [27], 1D CNN based on the multi-learning trick (MLT) [14], cascaded denoising CNN (Dn-CNN) [28], and a pre-trained deep CNN model with attention [29]. The learning features are the main attraction of different DL-based SER methods.…”

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

“…MLT‐DNet [72], which is based on one‐dimensional dilated CNNs where the model uses a multi‐learning technique to extract spatial salient emotional features and long‐term contextual dependencies from speech signals. The other baselines include FaceNet [73] takes spectrogram and waveform as input, HGFM [74] is a hierarchical‐grained feature model, DualNet [22] composed of an attention‐based BLSTM, The graph attention‐based GRU (GA‐GRU) [75], XGBoost [76] Dual Att‐BLSTM [77], 3D‐CNN + ASRNN [78], and AMS‐Net [11].…”

A Survey on Multi-modal Emotion Detection Techniques

“…Bubai et. al [53] presented Mel-spectrograms using the Conv-Cap module, and the remaining spectral characteristics from the input tensor were obtained using the Bi-GRU. Each module made use of the self-attention layer to identify the attention weight and preferentially concentrate on the best cues in order to produce high-level features.…”