Layer-wise Analysis of a Self-supervised Speech Representation Model

Pasad, Ankita; Chou, Ju-Chieh; Livescu, Karen

doi:10.48550/arxiv.2107.04734

Cited by 9 publications

(12 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, our ap- We conducted a layer-difference analysis study to determine which W2V2 layer contributes the most to the SER task. [20] examined the layer-specific information in W2V2 intermediate speech representations and found that various acoustic and linguistic properties are encoded in different layers. As such, we compared the hidden-state outputs from the first layer (initial embeddings), middle layer, and final layer and found a surprising difference in SER performance.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Li¹,

Bell²,

Lai³

2021

Preprint

View full text Add to dashboard Cite

Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Li¹,

Bell²,

Lai³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To get a finer-grained understanding of how linguistic representations evolve across these models, we turned away from the fMRI data and instead compared each layer directly to known linguistic features. Inspired by prior work in computer vision (Alain & Bengio, 2017), natural language processing (Ettinger et al, 2016;Shi et al, 2016) and speech (Pasad et al, 2021;Yang et al, 2021), we did this by linearly probing each layer's representations for spectral features (FBANK), spectrotemporal features, phoneme identity, and word identity.…”

Section: Probing Ssl Models For Linguistic Structurementioning

confidence: 99%

“…Our experiment comprised human participants passively listening to English-language narrative stories while their whole-brain fMRI BOLD activity was being recorded. Prior work on LM-based language encoding models has found performance differences across LM layers (Jain & Huth, 2018;Toneva & Wehbe, 2019), and layer-wise analyses of SSL models have shown that they capture different types of acoustic information (Pasad et al, 2021). Motivated by these findings, we built separate encoding models for each layer in four SSL models -APC, wav2vec, wav2vec 2.0 and HuBERT (Table 1).…”

Section: Introductionmentioning

confidence: 99%

“…Using autoregressive, contrastive, and masked prediction losses, SSL models learn powerful acoustic representations that are highly transferable across applications (Hsu et al, 2021;Schneider et al, 2019;Baevski et al, 2020). Much like language models, SSL models of speech have also been shown to capture higher-order linguistic structure without explicit supervision (Pasad et al, 2021;Yang et al, 2021). This can be attributed to their ability to integrate information over a broader context than traditional acoustic filters.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-supervised models of audio effectively explain human cortical responses to speech

Vaidya¹,

Jain²,

Huth³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new stateof-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models' ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.

show abstract

“…Secondly, a very recent study reveals that similar to supervised learning, SSL too gets biased to the domain from which the unlabelled data originates [11]. Thirdly, as SSL implicitly learns a language model and other semantic information through the tasks it is subjected to solve [12], the generalizability of these models is only to the extent where data from a similar language or phonetic structure is introduced to it at finetuning. Thus, as correctly pointed out by [13], SSL for speech suffers from the problems of scale, and SSL generalizability can be improved with more efficient training procedures.…”

Section: Introductionmentioning

confidence: 99%

PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations

Prasad¹,

Ghosh²,

Umesh³

2022

Preprint

View full text Add to dashboard Cite

While self-supervised speech representation learning (SSL) models serve a variety of downstream tasks, these models have been observed to overfit to the domain from which the unlabelled data originates. To alleviate this issue, we propose PADA (Pruning Assisted Domain Adaptation), and zero out redundant weights from models pre-trained on large amounts of outof-domain (OOD) data. Intuitively, this helps to make space for the target-domain ASR finetuning. The redundant weights can be identified through various pruning strategies which have been discussed in detail as a part of this work. Specifically, we investigate the effect of the recently discovered Task-Agnostic and Task-Aware pruning on PADA and propose a new pruning paradigm based on the latter, which we call Cross-Domain Task-Aware Pruning (CD-TAW). CD-TAW obtains the initial pruning mask from a well fine-tuned OOD model, which makes it starkly different from the rest of the pruning strategies discussed in the paper. Our proposed CD-TAW methodology achieves up to 20.6% relative WER improvement over our baseline when fine-tuned on a 2-hour subset of Switchboard data without language model (LM) decoding. Furthermore, we conduct detailed analysis to highlight the key design choices of our proposed method.

show abstract

Layer-wise Analysis of a Self-supervised Speech Representation Model

Cited by 9 publications

References 31 publications

Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Self-supervised models of audio effectively explain human cortical responses to speech

PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations

Contact Info

Product

Resources

About