Language Tags Matter for Zero-Shot Neural Machine Translation

Wu, Liwei; Cheng, Shanbo; Wang, Mingxuan; Li, Lei

doi:10.18653/v1/2021.findings-acl.264

Cited by 9 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This finding disagrees with that of Wang et al (2021), which we ascribe to various differences in model, data and optimization. Note that Wang et al (2021) adopted more aggressive data oversampling, didn't consider distant languages, proposed dedicated optimization with the source-side loss, used a different way to count model parameters, and designed different language tags for multilingual translation that could greatly affect zero-shot results (Wu et al, 2021). We leave the study of these differences to the future.…”

Section: Experiments For Cross-lingual Transfermentioning

confidence: 99%

Examining Scaling and Transfer of Language Model Architectures for Machine Translation

Zhang¹,

Ghorbani²,

Bapna³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, and improve greatly on zero-shot directions by facilitating the reduction of off-target translations.

show abstract

Section: Experiments For Cross-lingual Transfermentioning

confidence: 99%

Examining Scaling and Transfer of Language Model Architectures for Machine Translation

Zhang¹,

Ghorbani²,

Bapna³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Raganato et al [32] proposed an auxiliary supervised objective that explicitly aligns the words in language pairs to alleviate the off-target problem of zero-shot translation. Wu et al [33] empirically compared several language tag strategies and recommended adding the target language tag to the encoder for zero-shot translation. Gonzales et al [34] improved the zero-shot translation by performing language-specific subword segmentation and including the non-English pairs in English-centric training data.…”

Section: Zero-shot Neural Machine Translationmentioning

confidence: 99%

Rectifying Ill-Formed Interlingual Space: A Framework for Zero-Shot Translation on Modularized Multilingual NMT

Liao

2022

Mathematics

View full text Add to dashboard Cite

The multilingual neural machine translation (NMT) model can handle translation between more than one language pair. From the perspective of industrial applications, the modularized multilingual NMT model (M2 model) that only shares modules between the same languages is a practical alternative to the model that shares one encoder and one decoder (1-1 model). Previous works have proven that the M2 model can benefit from multiway training without suffering from capacity bottlenecks and exhibits better performance than the 1-1 model. However, the M2 model trained on English-centric data is incapable of zero-shot translation due to the ill-formed interlingual space. In this study, we propose a framework to help the M2 model form an interlingual space for zero-shot translation. Using this framework, we devise an approach that combines multiway training with a denoising autoencoder task and incorporates a Transformer attention bridge module based on the attention mechanism. We experimentally show that the proposed method can form an improved interlingual space in two zero-shot experiments. Our findings further extend the use of the M2 model for multilingual translation in industrial applications.

show abstract

HW-TSC’s Neural Machine Translation System for CCMT 2023

Wu,

Yu,

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Language Tags Matter for Zero-Shot Neural Machine Translation

Cited by 9 publications

References 28 publications

Examining Scaling and Transfer of Language Model Architectures for Machine Translation

Examining Scaling and Transfer of Language Model Architectures for Machine Translation

Rectifying Ill-Formed Interlingual Space: A Framework for Zero-Shot Translation on Modularized Multilingual NMT

HW-TSC’s Neural Machine Translation System for CCMT 2023

Contact Info

Product

Resources

About