An Investigation of End-to-End Models for Robust Speech Recognition

Abstract

End-to-end models for robust automatic speech recognition (ASR) have not been sufficiently well-explored in prior work. With end-to-end models, one could choose to preprocess the input speech using speech enhancement techniques and train the model using enhanced speech. Another alternative is to pass the noisy speech as input and modify the model architecture to adapt to noisy speech. A systematic comparison of these two approaches for end-to-end robust ASR has not been attempted before. We address this gap and present a detailed comparison of speech enhancement-based techniques and three different model-based adaptation techniques covering data augmentation, multi-task learning, and adversarial learning for robust ASR. While adversarial learning is the best-performing technique on certain noise types, it comes at the cost of degrading clean speech WER. On other relatively stationary noise types, a new speech enhancement technique outperformed all the model-based adaptation techniques. This suggests that knowledge of the underlying noise type can meaningfully inform the choice of adaptation technique.

Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASR

“…LibriSpeech: In order to facilitate a fair comparison with existing approaches, the data usage throughout experiments keeps exactly the same as that in [48]. Specifically, we utilize the LibriSpeech [49] real recorded and 1640 simulated noisy utterances, and the test set contains 1320 real recorded and 1320 simulated noisy utterances.…”

“…Comparison methods: The Baseline in [48] utilizes the Deepspeech2 model [53] for training on the LibriSpeech train-clean-100 dataset with a CTC objective function and evaluates on different test sets. For completeness, the timedomain DEMUCS as a front-end SE step in [48] will be compared, where the enhanced speech is directly used for ASR.…”

“…Comparison methods: The Baseline in [48] utilizes the Deepspeech2 model [53] for training on the LibriSpeech train-clean-100 dataset with a CTC objective function and evaluates on different test sets. For completeness, the timedomain DEMUCS as a front-end SE step in [48] will be compared, where the enhanced speech is directly used for ASR. The AvT method utilized in [48] introduces a gradient reversal layer in prior to the model classification layer, such that the learned speech representations can be noise-invariant.…”

“…For completeness, the timedomain DEMUCS as a front-end SE step in [48] will be compared, where the enhanced speech is directly used for ASR. The AvT method utilized in [48] introduces a gradient reversal layer in prior to the model classification layer, such that the learned speech representations can be noise-invariant. In addition, the proposed EW2 model will also be compared with the original wav2vec2.0 method [31].…”

Multiple Confidence Gates For Joint Training Of SE And ASR

“…Although the enhanced speech is greatly improved currently in human hearing [6][7][8][9], the enhanced feature distribution is changed also, which may be helpful for the hearing but not always necessarily beneficial for ASR. In order to make the ASR adaptable to the enhanced feature, the ASR can be retrained with the data processed by the enhanced model [10]. However, the performance of this method is severely affected by the SE effect.…”