Language-aware Interlingua for Multilingual Neural Machine Translation

Zhu, Changfeng; Yu, Heng; Cheng, Shanbo; Luo, Weihua

doi:10.18653/v1/2020.acl-main.150

Cited by 27 publications

(22 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing works on translation from multiple source languages into a single low resource language usually have at most 30 source languages (Gu et al, 2018;Zhou et al, 2018a;Zhu et al, 2020). They are limited within the same or close-by language families, or those with available data, or those chosen based on the researchers' intuitive discretion.…”

Section: Ranking Source Languagesmentioning

confidence: 99%

“…We focus on five challenges that are not addressed previously. Most multilingual transformer works that translate into low resource language limit their training data to available data in the same or close-by language families or the researchers' intuitive discretion; and are mostly limited to less than 30 languages (Gu et al, 2018;Zhou et al, 2018a;Zhu et al, 2020). Instead, we examine ways to pick useful source languages from 124 source languages in a principled fashion.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

Zhou¹,

Waibel²

2021

Proceedings of the Third Workshop on Computational Typology and Multilingual NLP

View full text Add to dashboard Cite

We translate a closed text that is known in advance into a severely low resource language by leveraging massive source parallelism. In other words, given a text in 124 source languages, we translate it into a severely low resource language using only ∼1,000 lines of low resource data without any external help. Firstly, we propose a systematic method to rank and choose source languages that are close to the low resource language. We call the linguistic definition of language family Family of Origin (FAMO), and we call the empirical definition of higher-ranked languages using our metrics Family of Choice (FAMC). Secondly, we build an Iteratively Pretrained Multilingual Orderpreserving Lexiconized Transformer (IPML) to train on ∼1,000 lines (∼3.5%) of low resource data. To translate named entities correctly, we build a massive lexicon table for 2,939 Bible named entities in 124 source languages, and include many that occur once and covers more than 66 severely low resource languages. Moreover, we also build a novel method of combining translations from different source languages into one. Using English as a hypothetical low resource language, we get a +23.9 BLEU increase over a multilingual baseline, and a +10.3 BLEU increase over our asymmetric baseline in the Bible dataset. We get a 42.8 BLEU score for Portuguese-English translation on the medical EMEA dataset. We also have good results for a real severely low resource Mayan language, Eastern Pokomchi.

show abstract

Section: Ranking Source Languagesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

Zhou¹,

Waibel²

2021

Proceedings of the Third Workshop on Computational Typology and Multilingual NLP

View full text Add to dashboard Cite

show abstract

“…And the usage of the language token style is widely accepted. Also, many subsequent works continuously explore new approaches in MNMT, such as parameter sharing (Blackwood et al, 2018;Wang et al, 2019b;Tan et al, 2019a), parameter generation (Platanios et al, 2018), knowledge distillation (Tan et al, 2019b), learning better representation (Wang et al, 2019a), massive training (Aharoni et al, 2019;Arivazhagan et al, 2019), interlingua (Zhu et al, 2020a), and adpater (Zhu et al, 2021). These works mainly utilize parallel data.…”

Section: Multilingual Neural Machine Translationmentioning

confidence: 99%

Multilingual Translation via Grafting Pre-trained Language Models

Sun

Wang

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pretrained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size 1 .

show abstract

“…Improving the consistency of semantic representations and alleviating the off-target issue (Zhang et al, 2020) are effective ways to improve the zeroshot translation quality (Al-Shedivat and Parikh, 2019;Arivazhagan et al, 2019;Zhu et al, 2020). The semantic representations of different languages should be close to each other to get better translation quality (Ding et al, 2017).…”

Section: Background and Notationsmentioning

confidence: 99%

Language Tags Matter for Zero-Shot Neural Machine Translation

Wu¹,

Cheng²,

Wang³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

Self Cite

View full text Add to dashboard Cite

Multilingual Neural Machine Translation (MNMT) has aroused widespread interest due to its efficiency. An exciting advantage of MNMT models is that they could also translate between unsupervised (zero-shot) language directions.Language tag (LT) strategies are often adopted to indicate the translation directions in MNMT. In this paper, we demonstrate that the LTs are not only indicators for translation directions but also crucial to zero-shot translation qualities. Unfortunately, previous work tends to ignore the importance of LT strategies. We demonstrate that a proper LT strategy could enhance the consistency of semantic representations and alleviate the off-target issue in zero-shot directions. Experimental results show that by ignoring the source language tag (SLT) and adding the target language tag (TLT) to the encoder, the zero-shot translations could achieve a +8 BLEU score difference over other LT strategies in IWSLT17, Europarl, TED talks translation tasks.

show abstract

Language-aware Interlingua for Multilingual Neural Machine Translation

Cited by 27 publications

References 12 publications

Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

Multilingual Translation via Grafting Pre-trained Language Models

Language Tags Matter for Zero-Shot Neural Machine Translation

Contact Info

Product

Resources

About