Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognition

Belinkov, Yonatan; Ali, Ahmed; Glass, James

doi:10.21437/interspeech.2019-2599

Cited by 56 publications

(65 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Finally, we hope to gain some understanding into why pretraining on ASR helps with AST, and specifically how the neural network representations change during pretraining and fine-tuning. We follow [35] and [10], who built diagnostic classifiers [36] to examine the representation of phonetic information in end-to-end ASR and AST systems, respectively. Unlike [10,35], who used non-linear classifiers, we use a linear classifier to predict phone labels from the internal representations of the trained ASR or AST model.…”

Section: Analyzing the Models' Representationsmentioning

confidence: 99%

“…We follow [35] and [10], who built diagnostic classifiers [36] to examine the representation of phonetic information in end-to-end ASR and AST systems, respectively. Unlike [10,35], who used non-linear classifiers, we use a linear classifier to predict phone labels from the internal representations of the trained ASR or AST model. Using a linear classifier allows us to make more precise claims: if the classifier performs better using the representation from a particular layer, we can say that layer represents the phonetic information in a more linearly separable way.…”

Section: Analyzing the Models' Representationsmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

Stoian

Bansal

Goldwater

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Previous work has shown that for low-resource source languages, automatic speech-to-text translation (AST) can be improved by pretraining an end-to-end model on automatic speech recognition (ASR) data from a high-resource language. However, it is not clear what factors-e.g., language relatedness or size of the pretraining datayield the biggest improvements, or whether pretraining can be effectively combined with other methods such as data augmentation. Here, we experiment with pretraining on datasets of varying sizes, including languages related and unrelated to the AST source language. We find that the best predictor of final AST performance is the word error rate of the pretrained ASR model, and that differences in ASR/AST performance correlate with how phonetic information is encoded in the later RNN layers of our model. We also show that pretraining and data augmentation yield complementary benefits for AST.

show abstract

Section: Analyzing the Models' Representationsmentioning

confidence: 99%

Section: Analyzing the Models' Representationsmentioning

confidence: 99%

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

Stoian

Bansal

Goldwater

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Although there is another choice of the CTC-based model for the A2C model as in [12], we adopt the attention-based model because character-level CTC models are more likely to misspell than attention-based models [10,11]. The character-level decoder can be connected to the arbitrary intermediate layer [23,24]. The overall loss function is the linear interpolation of the negative log-likelihood between the A2W and A2C models by a tunable parameter λ (0 ≤ λ ≤ 1):…”

Section: Multi-task Learning With Attention-based A2c Modelmentioning

confidence: 99%

Improving OOV Detection and Resolution with External Language Models in Acoustic-to-Word ASR

Inaguma

Mimura

Sakai

et al. 2018

2018 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Acoustic-to-word (A2W) end-to-end automatic speech recognition (ASR) systems have attracted attention because of an extremely simplified architecture and fast decoding. To alleviate data sparseness issues due to infrequent words, the combination with an acoustic-tocharacter (A2C) model is investigated. Moreover, the A2C model can be used to recover out-of-vocabulary (OOV) words that are not covered by the A2W model, but this requires accurate detection of OOV words. A2W models learn contexts with both acoustic and transcripts; therefore they tend to falsely recognize OOV words as words in the vocabulary. In this paper, we tackle this problem by using external language models (LM), which are trained only with transcriptions and have better linguistic information to detect OOV words. The A2C model is used to resolve these OOV words. Experimental evaluations show that external LMs have the effects of not only reducing errors but also increasing the number of detected OOV words, and the proposed method significantly improves performances in English conversational and Japanese lecture corpora, especially for out-of-domain scenario. We also investigate the impact of the vocabulary size of A2W models and the data size for training LMs. Moreover, our approach can reduce the vocabulary size several times with marginal performance degradation.Index Terms-End-to-end speech recognition, acoustic-toword, attention-based encoder-decoder, multi-task learning, recurrent neural network language model, OOV arXiv:1909.09993v1 [cs.CL]

show abstract

“…better comprehending the complex, highly nonlinear transformations inside the network. Previous research analyzing end-to-end ASR involves investigating the underlying phonetic representations learned in the course of training [2,3,4]. Interpretable filters with SincNet [5] is proposed and shown capable of removing noise after training.…”

Section: Introductionmentioning

confidence: 99%

What Does a Network Layer Hear? Analyzing Hidden Representations of End-to-End ASR Through Speech Synthesis

Yuan

Lee

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

End-to-end speech recognition systems have achieved competitive results compared to traditional systems. However, the complex transformations involved between layers given highly variable acoustic signals are hard to analyze. In this paper, we present our ASR probing model, which synthesizes speech from hidden representations of end-to-end ASR to examine the information maintain after each layer calculation. Listening to the synthesized speech, we observe gradual removal of speaker variability and noise as the layer goes deeper, which aligns with the previous studies on how deep network functions in speech recognition. This paper is the first study analyzing the end-to-end speech recognition model by demonstrating what each layer hears. Speaker verification and speech enhancement measurements on synthesized speech are also conducted to confirm our observation further.

show abstract

Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognition

Cited by 56 publications

References 25 publications

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

Improving OOV Detection and Resolution with External Language Models in Acoustic-to-Word ASR

What Does a Network Layer Hear? Analyzing Hidden Representations of End-to-End ASR Through Speech Synthesis

Contact Info

Product

Resources

About