ISNet: Individual Standardization Network for Speech Emotion Recognition

Fan, Weiquan; Xu, Xiangmin; Cai, Bolun; Xing, Xiaofen

doi:10.1109/taslp.2022.3171965

Cited by 32 publications

(18 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This facilitates the capturing of long-distance dependencies within audio signals, thereby improving the extraction of both local and high-level features, Table 1 The variation in computational efficiency and performance of TFC-SpeechFormer and Transformer using HCE across different datasets.The symbols "(+)" indicate improvement, while "(-)" indicates a decrease. IEMOCAP STC [30] 0.613 0.604 0.617 LSTM-GIN [31] 0.647 0.655 -CA-MSER [32] 0.698 0.711 -SpeechFormer [24] 0.629 0.645 -ISNet [33] 0.704 0.65 -DST [22] 0.718 0.736 -ShiftSER [34] 0.721 0.727 -TFC-SpeechFormer(Ours) 0.746 0.751 0.743 DAIC-WOZ FVTC-CNN [35] 0.735 0.656 0.64 Saidi [36] 0.68 0.68 0.68 EmoAudioNet [37] 0.732 0.649 0.653 Solieman [38] 0.66 0.615 0.61 SIMSIAM-S [39] 0.703 --TOAT [40] 0.717 0.429 0.48 SpeechFormer [24] 0.686 0.65 0.694 TFC-SpeechFormer(Ours) 0.762 0.701 0.714 consequently enhancing the feature representation capability. This, in turn, enables the model to better understand the emotional implications within speech signals.…”

Section: Experimental Results and Analysismentioning

confidence: 99%

TFC-SpeechFormer: Efficient Emotion Recognition Based on Deep Speech Analysis and Hierarchical Progressive Structures

Wei,

Zhang

2024

Preprint

View full text Add to dashboard Cite

Speech emotion recognition aims to automatically identify emotions in human speech, enabling machines to understand and engage emotional communication. In recent years, Transformers have demonstrated strong adaptability and significant effectiveness in speech recognition. However, Transformer models are proficient at capturing local features, struggle to extract fine-grained details, and often lead to computational redundancy, increasing time complexity. This paper proposes a Speech Emotion Recognition model named Temporal Fusion Convolution SpeechFormer (TFCS).The model comprises a Hybrid Convolutional Extractor (HCE) and multiple Temporal Fusion Convolution SpeechFormer Blocks (TFCSBs). HCE, consisting of an encoder and convolutional modules, enhances speech signals to capture local features and texture information, extracting frame-level features. TFCSBs utilize a newly proposed Temporal Fusion Convolution Module and Speech Multi-Head Attention Module to capture correlations between adjacent elements in speech sequences. TFCSBs utilize feature information captured by HCE to sequentially form frame, phoneme, word, and sentence structures, and integrates them to establish global and local relationships, enhancing local data capture and computational efficiency. Performance evaluation on the IEMOCAP and DAIC-WOZ datasets demonstrates the effectiveness of HCE in extracting fine-grained local speech features, with TFCS outperforming Transformer and other advanced models overall.

show abstract

Section: Experimental Results and Analysismentioning

confidence: 99%

TFC-SpeechFormer: Efficient Emotion Recognition Based on Deep Speech Analysis and Hierarchical Progressive Structures

Wei,

Zhang

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Our SpeechFormer++ with hand-crafted features outperforms STC [14] (0.645 vs. 0.613 in WA, 0.658 vs. 0.604 in UA and 0.649 vs. 0.617 in WF1) and achieves comparable results to LSTM-GIN [17] (0.645 vs. 0.647 in WA and 0.658 vs. 0.655 in UA) under the same experimental setup. SpeechFormer++ obtains inferior results compared to ISNet [15]. We suspect this is because ISNet is equipped with a carefully designed individual benchmark to alleviate the problem of interindividual emotion confusion.…”

Section: A Speech Emotion Recognition On Iemocap 1) Comparison To Tra...mentioning

confidence: 99%

SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing

Chen,

Xing,

et al. 2023

Preprint

View full text Add to dashboard Cite

Paralinguistic speech processing is important in addressing many issues, such as sentiment and neurocognitive disorder analyses. Recently, Transformer has achieved remarkable success in the natural language processing field and has demonstrated its adaptation to speech. However, previous works on Transformer in the speech field have not incorporated the properties of speech, leaving the full potential of Transformer unexplored. In this paper, we consider the characteristics of speech and propose a general structure-based framework, called SpeechFormer++, for paralinguistic speech processing. More concretely, following the component relationship in the speech signal, we design a unit encoder to model the intra-and inter-unit information (i.e., frames, phones, and words) efficiently. According to the hierarchical relationship, we utilize merging blocks to generate features at different granularities, which is consistent with the structural pattern in the speech signal. Moreover, a word encoder is introduced to integrate word-grained features into each unit encoder, which effectively balances fine-grained and coarse-grained information. SpeechFormer++ is evaluated on the speech emotion recognition (IEMOCAP & MELD), depression classification (DAIC-WOZ) and Alzheimer's disease detection (Pitt) tasks. The results show that SpeechFormer++ outperforms the standard Transformer while greatly reducing the computational cost. Furthermore, it delivers superior results compared to the state-of-the-art approaches.

show abstract

“…The majority (31/51, 60.8%) of the study items adopted a uni-modal approach with audio as the sole modality. A smaller number of studies used multi-modal approaches that included: (a) only audio and video (2/51, 3.9%), [45][46][47][48][49][50][51], (b) only audio and text (5/51, 9.8%), and (c) audio, video and text (12/51, 23.5%). Only one study (1.9%) utilized physiological signals for SERC [52].…”

Section: Characteristics Of the Included Studiesmentioning

confidence: 99%

“…ResNet18 [32]) or pre-trained transfer-learned feature extractors, e.g. Wav2vec [54]), here accounting for 25.5%(13/51) of total studies; (c) image transformations, summing to 19.6%(10/51) as yielded by advanced signal processing methods of raw waveforms, such as, spectrograms [48,55] or Mel-Frequency Cepstral Coefficients (MFCCs) [47,56]; (d) hybrid approaches, as combinations of two or three of the aforementioned options, here appearing in 25.5%(13/51) of study items. A trend towards deep learning based approaches in SERC can be observed after 2019, when standalone or hybrid deep learning emerged.…”

Section: Characteristics Of the Included Studiesmentioning

confidence: 99%

Speech Emotion Recognition in Conversations Using Artificial Intelligence: A Systematic Review and Meta-Analysis

Alhussein,

Ziogas,

Saleem

et al. 2023

Preprint

View full text Add to dashboard Cite

Purpose: Manifestations of emotion in social conversational interactions stand at a focal point in the rapidly growing affective computing area, with applications in healthcare, education and human-computer interaction. Artificial intelligence (AI) holds great potential in modelling the challenging dynamic nature of affect in speech conversation. In this paper, we analyze and criticize latest trends and open problems through a systematic review and multi-subgroup meta-analysis of AI approaches for emotion recognition in conversation (ERC). Methods: We adopt the PRISMA-DTA guidelines toward analysis of AI-driven speech ERC. A comprehensive database search through predefined query strings and selection criteria allowed for data extraction of essential diagnostic performance parameters. We analyze salient patterns related to methodological quality and risk of bias. Univariate random-effects models are then designed with a multi-subgroup perspective, centered around affective annotations models, while encompassing the ERC parameters of modalities, feature extraction and conversation style. Results: 51 studies were systematically reviewed for qualitative analysis, whereas 27 articles were included in the meta-analysis. Diagnostic test performance manifested with high heterogeneity, with intriguing insights regarding affective state annotation, input modality, feature extraction methods, and dataset conversation style. Conclusion: Our research contributes fine-grained insights as recommendations that tackle open-problems in ERC. While providing valuable information on diagnostic performance of AI in speech ERC, we underscore the imperative need for further advancements in annotations and models capable of handling diverse emotional expressions.

show abstract

ISNet: Individual Standardization Network for Speech Emotion Recognition

Cited by 32 publications

References 48 publications

TFC-SpeechFormer: Efficient Emotion Recognition Based on Deep Speech Analysis and Hierarchical Progressive Structures

TFC-SpeechFormer: Efficient Emotion Recognition Based on Deep Speech Analysis and Hierarchical Progressive Structures

SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing

Speech Emotion Recognition in Conversations Using Artificial Intelligence: A Systematic Review and Meta-Analysis

Contact Info

Product

Resources

About