Meta-Adapter: Efficient Cross-Lingual Adaptation With Meta-Learning

Abstract

Transfer learning from a multilingual model has shown favorable results on low-resource automatic speech recognition (ASR). However, full-model fine-tuning generates a separate model for every target language and is not suitable for deploying and maintaining in production. The key challenge lies in how to efficiently extend the pre-trained model with fewer parameters. In this paper, we propose to combine the adapter module with meta-learning algorithms to achieve high recognition performance under low-resource settings and improve the parameter-efficiency of the model. Extensive experiments show that our methods can achieve comparable or even superior recognition rates than the state-of-the-art baselines on low-resource languages, especially under very-low-resource conditions, with a significantly smaller model profile.

Exploiting Adapters for Cross-Lingual Low-Resource Speech Recognition

“…Besides learning the language-agnostic features, the optimization-based meta-learning approaches [13], [26] that aim to find a proper initialization for rapid adaptation have also been explored for cross-lingual ASR [9]. Hsu et al [8] proposed to apply the model-agnostic meta-learning (MAML) [13] as the pre-training method and achieved significant improvement over the conventional multilingual pretraining baseline.…”

“…Due to the limited training data in low-resource languages, direct re-training makes the model easily overfit. These problems make the transfer-based methods inefficient [9], [10]. Recently, the adapter module was proposed for parameter-efficient fine-tuning in multilingual or cross-lingual settings [9]- [11], which can mitigate overfitting.…”

“…These problems make the transfer-based methods inefficient [9], [10]. Recently, the adapter module was proposed for parameter-efficient fine-tuning in multilingual or cross-lingual settings [9]- [11], which can mitigate overfitting. Adapter is an add-on module to the encoder and decoder layers in Transformer that mainly composed of layer normalization and fully-connected layers.…”

“…In our previous work [9], we proposed MetaAdapter to learn general and transferable speech representations using model-agnostic meta-learning (MAML) [13] and achieved promising results on extremely low-resource languages. However, it is unclear whether MetaAdapter can handle the nonextreme cases where there are moderate training data.…”

“…• We show that these two algorithms can be integrated to achieve better performance with up to 3.55% relative improvement. This paper is substantially an extended version of our previously published paper [9] at ICASSP 2021. Compared to the previous version, we make heavy extensions as follows: (1) We propose a parallel new algorithm called SimAdapter for crosslingual ASR.…”