Extremely low-resource neural machine translation for Asian languages

Abstract

This paper presents a set of effective approaches to handle extremely low-resource language pairs for self-attention based neural machine translation (NMT) focusing on English and four Asian languages. Starting from an initial set of parallel sentences used to train bilingual baseline models, we introduce additional monolingual corpora and data processing techniques to improve translation quality. We describe a series of best practices and empirically validate the methods through an evaluation conducted on eight translation directions, based on state-of-the-art NMT approaches such as hyper-parameter search, data augmentation with forward and backward translation in combination with tags and noise, as well as joint multilingual training. Experiments show that the commonly used default architecture of self-attention NMT models does not reach the best results, validating previous work on the importance of hyper-parameter tuning. Additionally, empirical results indicate the amount of synthetic data required to efficiently increase the parameters of the models leading to the best translation quality measured by automatic metrics. We show that the best NMT models trained on large amount of tagged back-translations outperform three other synthetic data generation approaches. Finally, comparison with statistical machine translation (SMT) indicates that extremely low-resource NMT requires a large amount of synthetic parallel data obtained with back-translation in order to close the performance gap with the preceding SMT approach.

SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation

“…We used the fairseq framework [43] with the Transformer [60] architecture with 6 layer encoder (except for Filipino where 4 encoder layers were sufficient), 6 layer decoder and 1 attention head, decided through hyperparameter tuning as suggested by Rubino et al [48]. Dropout of 0.1 and label smoothing of 0.1 is used.…”

Inequalities and content moderation

“…Considerable technical constraints exist in training algorithmic technologies to detect harmful content in a wide range of languages. Martinus and Abbott (2019) underlined some of the primary obstacles in machine translation for African languages such as limited availability and discoverability. The lack of language training also comes from the difficulty in accessing resources in uncommon languages.…”

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

“…Monolingual data is often more freely available than bilingual data for low resource translation, as for small-domain translation. Data-centric approaches to low-resource language translation may therefore adapt to semi-synthetic datasets created by forward-or back-translation (Rubino et al, 2020;Karakanta et al, 2018). Even if this pseudo-parallel data is generated using a relatively weak low resource translation system, it still may be beneficial for further tuning that system (Currey & Heafield, 2019).…”