Context-Aware Fine-Tuning of Self-Supervised Speech Models

Shon, Suwon; Wu, Felix; Kim, Kwangyoun; Sridhar, Prashant; Livescu, Karen; Watanabe, Shinji

doi:10.1109/icassp49357.2023.10094687

Cited by 4 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following the results obtained on context at a speech turns level with the text modality, we incorporate the context from the previous or next turn of the target segment. Furthermore we implemented a novel way to enrich the yielded wav2vec2 embeddings through a dedicated auxiliary context module influenced by [19]. The auxiliary module is detailed in Fig.…”

Section: Contextual Exploration Of Acoustic Modalitymentioning

confidence: 99%

“…These models has shown strong results across various speech tasks benchmarks as highlighted in [21]. Our paper proposes context-aware fine-tuning, which utilizes surrounding speech segments during fine-tuning to improve performance on downstream speech tasks and enrich Transformer embeddings through the integration of auxiliary context module, as illustrated by [19] and by [15] with their emotionaware Transformer Emoformer.…”

Section: Introduction and Recent Workmentioning

confidence: 99%

“…In this work we propose a multi-scale hierarchical training system adapted to pre-trained standard attention models which are available by the French community. The proposed approach draws inspiration from recent work by [19]. We evaluate the impact of different types of contextual information for acoustic level and manual speech transcription.…”

Section: Introduction and Recent Workmentioning

confidence: 99%

“…Figure 5: Illustration of CCFTE Concatenation of Context Features with Target Embeddings, figure from[19] …”

mentioning

confidence: 99%

See 3 more Smart Citations

Investigating Transformer Encoders and Fusion Strategies for Speech Emotion Recognition in Emergency Call Center Conversations.

Deschamps-Berger

Lamel²,

Devillers³

2022

International Conference on Multimodal Interaction

View full text Add to dashboard Cite

The emotion detection technology to enhance human decision-making is an important research issue for real-world applications, but real-life emotion datasets are relatively rare and small. The experiments conducted in this paper use the CEMO, which was collected in a French emergency call center. Two pre-trained models based on speech and text were fine-tuned for speech emotion recognition. Using pre-trained Transformer encoders mitigates our data's limited and sparse nature. This paper explores the different fusion strategies of these modality-specific models. In particular, fusions with and without cross-attention mechanisms were tested to gather the most relevant information from both the speech and text encoders. We show that multimodal fusion brings an absolute gain of 4-9% with respect to either single modality and that the Symmetric multi-headed cross-attention mechanism performed better than late classical fusion approaches. Our experiments also suggest that for the real-life CEMO corpus, the audio component encodes more emotive information than the textual one.

show abstract

Section: Contextual Exploration Of Acoustic Modalitymentioning

confidence: 99%