What Does a Network Layer Hear? Analyzing Hidden Representations of End-to-End ASR Through Speech Synthesis

Li, Chung‐Yi; Yuan, Pei-Chieh; Lee, Hung-yi

doi:10.1109/icassp40776.2020.9054675

Cited by 27 publications

(17 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One notable exception is [17], which finds that the hidden layers of these networks outperform traditional features for a range of tasks, including speaker identification, emotion classification, and speechto-text. [18] found that the learned features become progressively more abstract at higher layers, normalizing for dimensions such as speaker, channel, and environmental conditions. In response to this sparse landscape, we asked the following questions:…”

Section: Introductionmentioning

confidence: 98%

Probing Acoustic Representations for Phonetic Properties

Ryant

Liberman

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Pre-trained acoustic representations such as wav2vec and De-CoAR have attained impressive word error rates (WER) for speech recognition benchmarks, particularly when labeled data is limited. But little is known about what phonetic properties these various representations acquire, and how well they encode transferable features of speech. We compare features from two conventional and four pre-trained systems in some simple frame-level phonetic classification tasks, with classifiers trained on features from one version of the TIMIT dataset and tested on features from another. All contextualized representations offered some level of transferability across domains, and models pre-trained on more audio data give better results; but overall, DeCoAR, the system with the simplest architecture, performs best. This type of benchmarking analysis can thus uncover relative strengths of various proposed acoustic representations.

show abstract

Section: Introductionmentioning

confidence: 98%

Probing Acoustic Representations for Phonetic Properties

Ryant

Liberman

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…In the audio domain such an approach using transfer learning was applied to study the capability of the layers of a pre-trained network to extract meaningful information for music classification and regression [51]. The role of each layer in an end-to-end speech recognition systems has been studied in [52]. The main idea is to synthesize speech signals from the hidden representations of each layer.…”

Section: Relation With Previous Workmentioning

confidence: 99%

An Interpretable Deep Learning Model for Automatic Sound Classification

et al. 2021

View full text Add to dashboard Cite

Deep learning models have improved cutting-edge technologies in many research areas, but their black-box structure makes it difficult to understand their inner workings and the rationale behind their predictions. This may lead to unintended effects, such as being susceptible to adversarial attacks or the reinforcement of biases. There is still a lack of research in the audio domain, despite the increasing interest in developing deep learning models that provide explanations of their decisions. To reduce this gap, we propose a novel interpretable deep learning model for automatic sound classification, which explains its predictions based on the similarity of the input to a set of learned prototypes in a latent space. We leverage domain knowledge by designing a frequency-dependent similarity measure and by considering different time-frequency resolutions in the feature space. The proposed model achieves results that are comparable to that of the state-of-the-art methods in three different sound classification tasks involving speech, music, and environmental audio. In addition, we present two automatic methods to prune the proposed model that exploit its interpretability. Our system is open source and it is accompanied by a web application for the manual editing of the model, which allows for a human-in-the-loop debugging approach.

show abstract

“…Elloumi et al (2018) use auxiliary classifiers to predict the underlying style of speech as being spontaneous or non-spontaneous and as having a native or non-native accent; their main task was to predict the performance of an ASR system on unseen broadcast programs. Analogous to saliency maps used to analyze images, Li et al (2020) propose reconstructing speech from the hidden representations at each layer using highway networks. Apart from ASR, analysis techniques have also been used with speaker embeddings for the task of speaker recognition (Wang et al, 2017).…”

Section: Analysis Of Asr Modelsmentioning

confidence: 99%

How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems

Prasad¹,

Jyothi²

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

In this work, we present a detailed analysis of how accent information is reflected in the internal representation of speech in an end-toend automatic speech recognition (ASR) system. We use a state-of-the-art end-to-end ASR system, comprising convolutional and recurrent layers, that is trained on a large amount of US-accented English speech and evaluate the model on speech samples from seven different English accents. We examine the effects of accent on the internal representation using three main probing techniques: a) Gradient-based explanation methods, b) Information-theoretic measures, and c) Outputs of accent and phone classifiers. We find different accents exhibiting similar trends irrespective of the probing technique used. We also find that most accent information is encoded within the first recurrent layer, which is suggestive of how one could adapt such an end-to-end model to learn representations that are invariant to accents.

show abstract

What Does a Network Layer Hear? Analyzing Hidden Representations of End-to-End ASR Through Speech Synthesis

Cited by 27 publications

References 16 publications

Probing Acoustic Representations for Phonetic Properties

Probing Acoustic Representations for Phonetic Properties

An Interpretable Deep Learning Model for Automatic Sound Classification

How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems

Contact Info

Product

Resources

About