Sequence-Based Multi-Lingual Low Resource Speech Recognition

Abstract

Techniques for multi-lingual and cross-lingual speech recognition can help in low resource scenarios, to bootstrap systems and enable analysis of new languages and domains. End-to-end approaches, in particular sequence-based techniques, are attractive because of their simplicity and elegance. While it is possible to integrate traditional multi-lingual bottleneck feature extractors as front-ends, we show that end-to-end multi-lingual training of sequence models is effective on context independent models trained using Connectionist Temporal Classification (CTC) loss. We show that our model improves performance on Babel languages by over 6% absolute in terms of word/phoneme error rate when compared to mono-lingual systems built in the same setting for these languages. We also show that the trained model can be adapted cross-lingually to an unseen language using just 25% of the target data. We show that training on multiple languages is important for very low resource cross-lingual target scenarios, but not for multi-lingual testing scenarios. Here, it appears beneficial to include large well prepared datasets.

Unsupervised Pretraining Transfers Well Across Languages

“…Closer to our work, several works have shared the parameters of a neural network encoder, using feedforward networks [3,1,2] or LSTM [11]. The model is then finetuned on the target low-resource language to fit its specificities [12]. The sampling of the languages during the pre-training can focus on languages related to the targeted language [11].…”

Multilingual Speech Recognition with Corpus Relatedness Sampling

“…Multilingual speech recognition has explored various models to share parameters across languages in different ways. For example, parameters can be shared by using posterior features from other languages [6], applying the same GMM components across different HMM states [7], training shared hidden layers in DNNs [3,4] or LSTM [5], using language independent bottleneck features [8,9]. Some models only share their hidden layers, but use separate output layers to predict their phones [3,4].…”

“…Our purpose here is to compute the embedding ei for each corpus Ci where ei is expected to encode information about its corpus Ci. Those embeddings can be jointly trained with the standard multilingual model [5]. First, the embedding matrix E for all corpora is initialized, the i-th row of E is corresponding to the embedding ei of the corpus Ci.…”

“…Every utterance in the corpora is firstly re-sampled into 8000Hz, and then we extract 40 dimension MFCCs features from each audio. We use a recent multilingual CTC model as our acoustic architecture [5]: The architecture is a 6 layer bidirectional LSTM model with 320 cells in each layer. We use this architecture for both the baseline models and the proposed model.…”

Meta Learning for End-To-End Low-Resource Speech Recognition

“…The model structure is designed to learn an encoder to extract language-independent representations to build a better acoustic model from many source languages. The success of "language-independent" features to improve ASR performance compared to monolingual training has been shown in many recent works [7,8,9]. Besides directly training the model with all the source languages, there are various variants of MultiASR approaches.…”