Multilingual Speech Recognition with Corpus Relatedness Sampling

Abstract

Multilingual acoustic models have been successfully applied to low-resource speech recognition. Most existing works have combined many small corpora together, and pretrained a multilingual model by sampling from each corpus uniformly. The model is eventually fine-tuned on each target corpus. This approach, however, fails to exploit the relatedness and similarity among corpora in the training set. For example, the target corpus might benefit more from a corpus in the same domain or a corpus from a close language. In this work, we propose a simple but useful sampling strategy to take advantage of this relatedness. We first compute the corpus-level embeddings and estimate the similarity between each corpus. Next we start training the multilingual model with uniform-sampling from each corpus at first, then we gradually increase the probability to sample from related corpora based on its similarity with the target corpus. Finally the model would be fine-tuned automatically on the target corpus. Our sampling strategy outperforms the baseline multilingual model on 16 low-resource tasks. Additionally, we demonstrate that our corpus embeddings capture the language and domain information of each corpus.

Unsupervised Pretraining Transfers Well Across Languages

“…Burget et al [10] shares the parameters of a Gaussian Mixture Model. Closer to our work, several works have shared the parameters of a neural network encoder, using feedforward networks [3,1,2] or LSTM [11]. The model is then finetuned on the target low-resource language to fit its specificities [12].…”

“…The model is then finetuned on the target low-resource language to fit its specificities [12]. The sampling of the languages during the pre-training can focus on languages related to the targeted language [11]. Another approach is to encourage a language-independent encoder with an adversarial loss [13].…”

Towards Zero-Shot Learning for Automatic Phonemic Transcription

“…We train the acoustic model with stochastic gradient descent, using a learning rate of 0.005. In each iteration, we apply the uniform sampling (Li et al 2019): first randomly select a corpus from the entire training set, and then randomly choose one batch from that corpus. Our baseline model is the multilingual acoustic model with a shared phoneme inventory.…”

Phone Distribution Estimation for Low Resource Languages

“…We associate languages from two datasets with ISO639-3 code and extract 676 languages. To recognize phones of various languages, a language-independent phone recognition model is required because any language-dependent systems could only discover phones of that specific language [16]. Additionally, languagedependent systems could not distinguish allophones which might be crucial in other languages [17].…”