Learning Speech Emotion Representations in the Quaternion Domain

Guizzo, Eric; Weyde, Tillman; Scardapane, Simone; Comminiello, Danilo

doi:10.1109/taslp.2023.3250840

Cited by 16 publications

(8 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, this database is often used in research on emotional speech [22,23] and is widely available. Secondly, as presented by the authors of [24], this database enables the highest efficiency of classification (88.47% [25]) compared to other databases, such as RAVDESS (87.5% [15,26]) or IEMOCAP (75.60% [27,28]). However, unlike the TESS [29] database, which is classified with an accuracy of 99.6% [30] contains longer utterances, not just words with the same prefix.…”

Section: Audio Datamentioning

confidence: 93%

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Smietanka,

Maka

2023

Journal of Artificial Intelligence and Soft Computing Research

View full text Add to dashboard Cite

The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.

show abstract

Section: Audio Datamentioning

confidence: 93%

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Smietanka,

Maka

2023

Journal of Artificial Intelligence and Soft Computing Research

View full text Add to dashboard Cite

show abstract

“…For the classifier, we opted to use a Convolutional Neural Network (CNN) as a deep learning model, as it has shown good performance in audio classification and speech recognition tasks [23]. Some papers in the literature that use CNN for the Speech Emotion Recognition (SER) problem can be seen in [3], [4], [8], [24], [25].…”

Section: Deep Learningmentioning

confidence: 99%

“…The choice of algorithm is crucial for successful recognition, as different algorithms have varying degrees of precision and efficiency. The choice of database is also essential, as the data quality used for training and testing can directly affect the system's ability to recognize emotions [4]. Additionally, the feature extraction technique is a critical factor, as selecting relevant features for the recognition task is essential for obtaining good results.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Speech Emotion Recognition:Evaluating Combinations of Databases, Data Augmentation, and Feature Extraction Methods

Ottoni,

Cerqueira

2023

Anais Do XVI Congresso Brasileiro De Inteligência Computacional

View full text Add to dashboard Cite

Speech emotion recognition is a challenging and essential task with numerous applications in human-computer interaction, healthcare, and entertainment. However, achieving high accuracy in this task is complicated by the need to select the best combination of machine learning algorithms, databases, data augmentation techniques, and feature extraction methods. This paper discusses the difficulty of choosing appropriate combinations of these factors and proposes a methodology to address this challenge. The proposed method evaluates the performance of various combinations of databases, data augmentation techniques, and feature extraction methods to determine the most effective approach for speech emotion recognition. The paper also presents a convolutional neural network to classify the emotions of happiness, sadness, fear, anger, surprise, disgust, and neutral. The results showed that the optimal combination proposed, with 94% accuracy, uses the combined RAVDESS and TESS databases, using data augmentation with noise, stretch, and pitch, and using MFCC to extract the characteristics of the audios.

show abstract

“…In recent studies on quaternion, researchers have achieved remarkable results by employing quaternion convolutional network (QCNN) for processing speech signals [24,25]. Quaternion, as extensions of hypercomplex numbers, enables the exploration and preservation of underlying connections within data information through quaternion convolution.…”

Section: Introductionmentioning

confidence: 99%

Multi-scale quaternion CNN and BiGRU with cross self-attention feature fusion for fault diagnosis of bearing

Liu,

Zhang,

Tan

et al. 2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

In recent years, deep learning has led to significant advances in bearing fault diagnosis (FD). Most techniques aim to achieve greater accuracy. However, they are sensitive to noise and lack robustness, resulting in insufficient domain adaptation and anti-noise ability. The comparison of studies reveals that giving equal attention to all features does not differentiate their significance. In this work, we propose a novel FD model by integrating multi-scale quaternion convolutional neural network (MQCNN), bidirectional gated recurrent unit (BiGRU), and cross self-attention feature fusion (CSAFF). We have developed innovative designs in two modules, namely MQCNN and CSAFF. Firstly, MQCNN applies quaternion convolution to multi-scale architecture for the first time, aiming to extract the rich hidden features of the original signal from multiple scales. Then, the extracted multi-scale information is input into CSAFF for feature fusion, where CSAFF innovatively incorporates cross self-attention mechanism to enhance discriminative interaction representation within features. Finally, BiGRU captures temporal dependencies while a softmax layer is employed for fault classification, achieving accurate FD. To assess the efficacy of our approach, we experiment on three public datasets (CWRU, MFPT, and Ottawa) and compare it with other excellent methods. The results confirm its state-of-the-art, which the average accuracies can achieve up to 99.99%, 100%, and 99.21% on CWRU, MFPT, and Ottawa datasets. Moreover, we perform practical tests and ablation experiments to validate the efficacy and robustness of the proposed approach.

show abstract

Learning Speech Emotion Representations in the Quaternion Domain

Cited by 16 publications

References 73 publications

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Optimizing Speech Emotion Recognition:Evaluating Combinations of Databases, Data Augmentation, and Feature Extraction Methods

Multi-scale quaternion CNN and BiGRU with cross self-attention feature fusion for fault diagnosis of bearing

Contact Info

Product

Resources

About