A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation

Abstract

Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, lowresource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pretrained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to finetune large pre-trained models on small quantities of high-quality translation data.

Building Text and Speech Benchmark Datasets and Models for Low‐Resourced East African Languages: Experiences and Lessons

“…Masakhane has created and publicly released several MT datasets and baseline models. 1 The most notable is MAFAND-MT, a news domain parallel corpus for 16 African languages [26]. Other efforts under Masakhane have been translating Edoid languages, MT from Fon to French [27], MT for Nigerian Pidgin [28], and many others.…”

“…Linguists from the languagespeaking communities led by the language coordinator provided quality control through discussion and fixtures of problematic translations. They also performed multiple checks to find and correct the misspellings in the dataset, which is a similar approach in other translation projects [26]. Each language had a lead responsible for ensuring the quality and approval of the final translations.…”

DaGATN: A Type of Machine Reading Comprehension Based on Discourse-Apperceptive Graph Attention Networks

“…Its encoding layer employs a self-attention mechanism and significantly improved performance compared to the RNN method. Subsequently, an increasing number of NLP tasks use methods based on pre-trained models, including named entity recognition [9][10][11], machine translation [12,13], and machine reading comprehension [14][15][16][17][18].…”