Speech Emotion Recognition with Global-Aware Fusion on Multi-Scale Feature Representation

Abstract

Speech Emotion Recognition (SER) is a fundamental task to predict the emotion label from speech data. Recent works mostly focus on using convolutional neural networks (CNNs) to learn local attention map on fixed-scale feature representation by viewing time-varied spectral features as images. However, rich emotional feature at different scales and important global information are not able to be well captured due to the limits of existing CNNs for SER. In this paper, we propose a novel GLobal-Aware Multi-scale (GLAM) neural network 1 to learn multi-scale feature representation with global-aware fusion module to attend emotional information. Specifically, GLAM iteratively utilizes multiple convolutional kernels with different scales to learn multiple feature representation. Then, instead of using attention-based methods, a simple but effective global-aware fusion module is applied to grab most important emotional information globally. Experiments on the benchmark corpus IEMOCAP over four emotions demonstrates the superiority of our proposed model with 2.5% to 4.5% improvements on four common metrics compared to previous state-of-the-art approaches.

A Residual Multi-Scale Convolutional Transformer Network with Chunk-level Log-Mel Spectrograms for Speech Emotion Recognition

“…After acquiring the 3D log-mel spectrogram, inspired by [57], we present a Residual Multi-Scale Covolutional Neural Network (RMSCNN) structure, which is shown in Figure 2b. We create two distinct convolutional kernels (1,3) and (3,1) to convolve the log-mel spectrogram before feeding it into the RMSCNN block.…”

“…By employing several scales and receptive fields of various sizes, the collection of more accurate emotional information is encouraged. In contrast to [57], we set a skip connection, by this creating a residual structure after the output of both convolutional layers; the same input and output channel numbers are maintained. The output feature maps are then concatenated along the channel dimension, and the BN and ReLU layers are added in the next step, as shown in the upper part of Figure 2c.…”

“…The Convolutional Transformer (CTx) [66] is a well-known model that consists of a group of conventional convolutional blocks connected with a Transformer block. The GLAM model [57] includes a global-aware block with two linear layers and a GeLU activation function for improving cross-scale feature communication, as well as a residual multi-scale convolutional neutral network topology that is quite similar to that presented in this study. After GeLU projection, the output is divided along the feature dimension, while the convolutional layer maintains the original channel size.…”

Speech Emotion Recognition Using RA-Gmlp Model on Time–Frequency Domain Features Extracted by TFCM

“…Jiale Yan [17] introduced TT-MLP using tensor decomposition to compress deep MLPs, thus reducing the amount of training parameters. Wenjing Zhu [18] utilized GMLP components and the global attention module (Glam) to achieve good results. However, speech emotional information is not uniformly distributed across speech features, and the aforementioned MLP structures did not consider how to focus the model on the required information.…”

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition

“…It uses deep neural networks to generate probability distributions of emotional states and construct discourse-level features, which are then fed into an Extreme Learning Machine (ELM) to identify discourse-level emotions. Zhu and Li ( 2022 ) used a global-aware fusion module to capture the most important emotional information across various scales. On the other hand, transfer learning is extensively implemented in the SER field.…”