Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition

Abstract

Low-resource automatic speech recognition (ASR) is challenging, as the low-resource target language data cannot well train an ASR model. To solve this issue, meta-learning formulates ASR for each source language into many small ASR tasks and meta-learns a model initialization on all tasks from different source languages to access fast adaptation on unseen target languages. However, for different source languages, the quantity and difficulty vary greatly because of their different data scales and diverse phonological systems, which leads to task-quantity and task-difficulty imbalance issues and thus a failure of multilingual meta-learning ASR (MML-ASR). In this work, we solve this problem by developing a novel adversarial meta sampling (AMS) approach to improve MML-ASR. When sampling tasks in MML-ASR, AMS adaptively determines the task sampling probability for each source language. Specifically, for each source language, if the query loss is large, it means that its tasks are not well sampled to train ASR model in terms of its quantity and difficulty and thus should be sampled more frequently for extra learning. Inspired by this fact, we feed the historical task query loss of all source language domain into a network to learn a task sampling policy for adversarially increasing the current query loss of MML-ASR. Thus, the learnt task sampling policy can master the learning situation of each language and thus predicts good task sampling probability for each language for more effective learning. Finally, experiment results on two multilingual datasets show significant performance improvement when applying our AMS on MML-ASR, and also demonstrate the applicability of AMS to other low-resource speech tasks and transfer learning ASR approaches. Our codes are available at: https://github.com/iamxiaoyubei/AMS.

Code-Switching in Automatic Speech Recognition: The Issues and Future Directions

“…Bilingual and multilingual recognition without CS is where the speech recognition system is not expected to switch from one language to another during the recognition process. These bilingual and multilingual systems usually use the existing databases for well-resourced languages [27,28,35,37,38,[44][45][46]57] and are self-developed for under-resourced languages [31,32,40].…”

“…Individuals that can speak more than one language (e.g., bilingual or multilingual) CS or mix their languages when communicating with others [3,5,[27][28][29][30][31][32][33][34][35][36][37][38][39]. CS is common in bilingual and multilingual communities (different cultures and language backgrounds) [40].…”

“…However, researchers may ignore the underlying task imbalance issues when sampling tasks from CS languages, which can lead to unsatisfactory performance by the ASR systems. First, different languages have different training data scales, so the basic task quantity for each language domain will be grossly different, leading to the task-quantity imbalance [38,53]. On top of that, as different languages have different phonological systems, the tasks taken from these languages have different recognition difficulties, causing the task-difficulty imbalance [38].…”

Multi-accent Speech Separation with One Shot Learning

“…(4). This is probably due to the task-difficulty imbalance issue described in (Xiao et al, 2020), perhaps some speakers with special accents may be hard to separate. Thus, in the future, we will try to solve this problem with meta sampling methods mentioned in (Xiao et al, 2020).…”

“…(Winata et al, 2020) applied metatransfer learning on code-switched speech recognition. (Xiao et al, 2020; applied meta-learning to solve the multilingual lowresource speech recognition problem. (Winata et al, 2019) also used MAML to adapt models to unseen accents on speech recognition.…”