Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

Abstract

We translate a closed text that is known in advance into a severely low resource language by leveraging massive source parallelism. In other words, given a text in 124 source languages, we translate it into a severely low resource language using only ∼1,000 lines of low resource data without any external help. Firstly, we propose a systematic method to rank and choose source languages that are close to the low resource language. We call the linguistic definition of language family Family of Origin (FAMO), and we call the empirical definition of higher-ranked languages using our metrics Family of Choice (FAMC). Secondly, we build an Iteratively Pretrained Multilingual Orderpreserving Lexiconized Transformer (IPML) to train on ∼1,000 lines (∼3.5%) of low resource data. To translate named entities correctly, we build a massive lexicon table for 2,939 Bible named entities in 124 source languages, and include many that occur once and covers more than 66 severely low resource languages. Moreover, we also build a novel method of combining translations from different source languages into one. Using English as a hypothetical low resource language, we get a +23.9 BLEU increase over a multilingual baseline, and a +10.3 BLEU increase over our asymmetric baseline in the Bible dataset. We get a 42.8 BLEU score for Portuguese-English translation on the medical EMEA dataset. We also have good results for a real severely low resource Mayan language, Eastern Pokomchi.

Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages

“…We take the most centered translation for every sentence, max i j S ij , to build the combined translation output. The expectation of the combined score is higher than that of any of the source languages (Zhou and Waibel, 2021).…”

“…We train our models using a state-of-the-art multilingual transformer by adding language labels to each source sentence (Johnson et al, 2017;Ha et al, 2016;Zhou et al, 2018a,b). We borrow the order-preserving named entity translation method by replacing each named entity with __NEs (Zhou et al, 2018b) using a multilingual lexicon table that covers 124 source languages and 2,939 named entities (Zhou and Waibel, 2021). For example, the sentence "Somchai calls Juan" is transformed to "__opt_src_en __opt_tgt_ca __NE0 calls __NE1" to translate to Chuj.…”

“…For example, the sentence "Somchai calls Juan" is transformed to "__opt_src_en __opt_tgt_ca __NE0 calls __NE1" to translate to Chuj. We use families of close-by languages constructed by ranking 124 source languages by distortion measure (FAMD), performance measure (FAMP) and linguistic family (FAMO + ); the distortion measure ranks languages by decreasing probability of zero distortion, while the performance measure incorporates an additional probability of fertility equalling one (Zhou and Waibel, 2021). Using families constructed, we pretrain our model first on the whole text of nearby languages, then we train on the ∼1,000 lines of low resource data and the corresponding lines in other languages in a multi-source multi-target fashion.…”

“…Using families constructed, we pretrain our model first on the whole text of nearby languages, then we train on the ∼1,000 lines of low resource data and the corresponding lines in other languages in a multi-source multi-target fashion. We finally train on the ∼1,000 lines in a multi-source single-target fashion (Zhou and Waibel, 2021). We combine translations of all source languages into one.…”

“…Some use data selection for active learning (Eck et al, 2005). Some use as few as ∼4,000 lines (Lin et al, 2020;Qi et al, 2018) and ∼1,000 lines (Zhou and Waibel, 2021) of data. Some do not use low resource data (Neubig and Hu, 2018;Karakanta et al, 2018).…”