Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP

Abstract

Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacherstudent knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers' knowledge into a single multilingual student. Our method outperforms standard training methods in lowresource languages and retains performance on high-resource languages. 1 5 6 See Section D.2 in the Appendix for more information.