Learning Noise Invariant Features Through Transfer Learning For Robust End-to-End Speech Recognition

Zhang, Shucong; Do, Cong-Thanh; Doddipatla, Rama; Renals, Steve

doi:10.1109/icassp40776.2020.9053169

Cited by 16 publications

(12 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…T/S learning was used to adapt a clean-trained E2E model to a noisy environment [112]. Noise-invariant feature was learned to improve robustness in [312,313]. Data augmentation [314] is another very effective way to expose more testing environments to E2E models during training.…”

Section: Miscellaneous Topicsmentioning

confidence: 99%

Recent Advances in End-to-End Automatic Speech Recognition

Li¹

2021

Preprint

View full text Add to dashboard Cite

Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR). While E2E models achieve the state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid models are still used in a large proportion of commercial ASR systems at the current time. There are lots of practical factors that affect the production model deployment decision. Traditional hybrid models, being optimized for production for decades, are usually good at these factors. Without providing excellent solutions to all these factors, it is hard for E2E models to be widely commercialized. In this paper, we will overview the recent advances in E2E models, focusing on technologies addressing those challenges from the industry's perspective.

show abstract

Section: Miscellaneous Topicsmentioning

confidence: 99%

Recent Advances in End-to-End Automatic Speech Recognition

Li¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this work, this simple strategy is used to show that the model can benefit from the unsupervised pre-training. There are more complex strategies for model fine-tuning or transfer learning in the literature, e.g., freezing or replacing layers, and adjusting the learning rate [25,26], which can be explored in our future work.…”

Section: Fine-tuningmentioning

confidence: 99%

Teacher-Student MixIT for Unsupervised and Semi-Supervised Speech Separation

Zhang¹,

Zorilă²,

Doddipatla³

et al. 2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce a novel semi-supervised learning framework for end-to-end speech separation. The proposed method first uses mixtures of unseparated sources and the mixture invariant training (MixIT) criterion to train a teacher model. The teacher model then estimates separated sources that are used to train a student model with standard permutation invariant training (PIT). The student model can be fine-tuned with supervised data, i.e., paired artificial mixtures and clean speech sources, and further improved via model distillation. Experiments with single and multi channel mixtures show that the teacher-student training resolves the over-separation problem observed in the original MixIT method. Further, the semisupervised performance is comparable to a fully-supervised separation system trained using ten times the amount of supervised data.

show abstract

“…In general, the reported experimental results do not indicate that these training methods surpass joint training. Belilovsky et al [18] and Nokland et al [19] Zhang et al [20] propose to use the classifier trained on the clean data to train the feature extractor on noisy data, so the noisy feature extractor is forced to learn features which fit the clean classifier and thus the method has the effect of denoising. In this work we show the classifier is transferable not only across different datasets but also within the same dataset.…”

Section: Related Workmentioning

confidence: 99%

Train Your Classifier First: Cascade Neural Networks Training from Upper Layers to Lower Layers

Zhang

Doddipatla

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same dataset leads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for highquality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

show abstract

Learning Noise Invariant Features Through Transfer Learning For Robust End-to-End Speech Recognition

Cited by 16 publications

References 26 publications

Recent Advances in End-to-End Automatic Speech Recognition

Recent Advances in End-to-End Automatic Speech Recognition

Teacher-Student MixIT for Unsupervised and Semi-Supervised Speech Separation

Train Your Classifier First: Cascade Neural Networks Training from Upper Layers to Lower Layers

Contact Info

Product

Resources

About