Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection

Abstract

With the prominence of large pretrained language models, low-resource languages are rarely modelled monolingually and become victims of the "curse of multilinguality" in massively multilingual models. Recently, Afri-BERTa showed that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. Here we extend this direction, focusing on the use of related languages. We propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. We test our hypothesis on the Niger-Congo family and its Bantu and Volta-Niger sub-families, pretraining models with data solely from Niger-Congo languages and finetuning on 4 downstream tasks: NER, partof-speech tagging, sentiment analysis and text classification. We find that models trained on genetically related languages achieve equal performance on downstream tasks in low-resource languages despite using less training data. We recommend selecting training data based on language-relatedness when pretraining language models for low-resource languages.

Embedding Structure Matters: Comparing Methods to Adapt Multilingual Vocabularies to New Languages

“…However, Wu and Dredze (2020) show that these massively multilingual models still underperform on lower-resource languages. Recent efforts to cover these languages instead pre-train models that are specialized to specific languages or language families (Ogueji et al, 2021;Ogunremi et al, 2023). These approaches nonetheless require training a new model from scratch and do not leverage transferable information in existing models.…”

Multilingual BERT has an Accent: Evaluating English Influences on Fluency in Multilingual Models

“…Our work builds off of a long literature on multilingual evaluation which has until now mostly focused on downstream classification tasks (Conneau et al, 2018;Ebrahimi et al, 2022;Clark et al, 2020;Liang et al, 2020;Hu et al, 2020;Raganato et al, 2020;Li et al, 2021). With the help of these evaluation methods, research has pointed out the problems for both high-and lowresource languages that come with adding many languages to a single model (Wang et al, 2020;Turc et al, 2021;Lauscher et al, 2020, inter alia), and proposed methods for more equitable models (Ansell et al, 2022;Pfeiffer et al, 2022;Ogueji et al, 2021;Ògúnrè . mí and Manning, 2023;Virtanen et al, 2019;Liang et al, 2023, inter alia).…”