Joint Unsupervised and Supervised Training for Multilingual ASR

Abstract

Self-supervised training has shown promising gains in pretraining models and facilitating the downstream finetuning for speech recognition, like multilingual ASR. Most existing methods adopt a 2-stage scheme where the self-supervised loss is optimized in the first pretraining stage, and the standard supervised finetuning resumes in the second stage. In this paper, we propose an end-to-end (E2E) Joint Unsupervised and Supervised Training (JUST) method to combine the supervised RNN-T loss and the self-supervised contrastive and masked language modeling (MLM) losses. We validate its performance on the public dataset Multilingual LibriSpeech (MLS), which includes 8 languages and is extremely imbalanced. On MLS, we explore (1) JUST trained from scratch, and (2) JUST finetuned from a pretrained checkpoint. Experiments show that JUST can consistently outperform other existing state-of-the-art methods, and beat the monolingual baseline by a significant margin, demonstrating JUST's capability of handling low-resource languages in multilingual ASR. Our average WER of all languages outperforms average monolingual baseline by 33.3%, and the state-of-the-art 2-stage XLSR by 32%. On low-resource languages like Polish, our WER is less than half of the monolingual baseline and even beats the supervised transfer learning method which uses external supervision.

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

“…Interestingly, with more finetune data, BEST-RQ perform even better than w2v-BERT, especially for pt and pl. Our results also comparable with previously stateof-the-art results in (Bai et al, 2021) which conduct joint training for multilingual ASR.…”

“…As our self-supervised learning algorithm eliminates the requirement of representation learning through applying a random-projection quantizer, it is crucial to understand the representation quality of this quantizer and how the quality of the quantization affect the self-supervised learning. We (Bai et al, 2021) 6.6 4.3 9.9 5.0 3.8 9.1 14.6 8.1 7.8 JUST (Bai et al, 2021) (co-train) 6.5 4.1 9.5 5.2 3.7 8.8 8.0 6.6 6.5 w2v-BERT (0.6B) 5.5 4.3 10.9 5.6 4.5 10.1 13.4 11.2 8.2 BEST-RQ (Ours, 0.6B) 6.8 4.1 9.7 5.0 4.9 7.4 9.4 5.2 6.6…”