Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation

Ji, Baijun; Zhang, Zhirui; Duan, Xiangyu; Zhang, Min; Chen, Boxing; Luo, Weihua

doi:10.1609/aaai.v34i01.5341

Cited by 47 publications

(48 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different loss functions such as cosine distance [4], Euclidean distance [113], and correlation distance [124] have been shown to be beneficial in reducing the source/pivot divergence. Ji et al [69] proposed to use pre-trained cross-lingual encoders trained using multilingual MLM, XLM, and BRLM objectives to obtain language-invariant encoder representations. Sen et al [129] used denoising autoencoding and back-translation to obtain languageinvariant encoder representations.…”

Section: :19mentioning

confidence: 99%

A Survey of Multilingual Neural Machine Translation

2020

View full text Add to dashboard Cite

We present a survey on multilingual neural machine translation (MNMT), which has gained a lot of traction in recent years. MNMT has been useful in improving translation quality as a result of translation knowledge transfer (transfer learning). MNMT is more promising and interesting than its statistical machine translation counterpart, because end-to-end modeling and distributed representations open new avenues for research on machine translation. Many approaches have been proposed to exploit multilingual parallel corpora for improving translation quality. However, the lack of a comprehensive survey makes it difficult to determine which approaches are promising and, hence, deserve further exploration. In this article, we present an indepth survey of existing literature on MNMT. We first categorize various approaches based on their central use-case and then further categorize them based on resource scenarios, underlying modeling principles, coreissues, and challenges. Wherever possible, we address the strengths and weaknesses of several techniques by comparing them with each other. We also discuss the future directions for MNMT. This article is aimed towards both beginners and experts in NMT. We hope this article will serve as a starting point as well as a source of new ideas for researchers and engineers interested in MNMT.

show abstract

Section: :19mentioning

confidence: 99%

A Survey of Multilingual Neural Machine Translation

2020

View full text Add to dashboard Cite

show abstract

“…While almost all encoders are pretrained without explicit crosslingual objective, i.e. enforcing similar words from different languages have similar representation, improvements can be attained through the use of explicit cross-lingually linked data during pretraining, such as bitexts (Conneau and Lample, 2019;Huang et al, 2019;Ji et al, 2019) and dictionaries . As with cross-lingual embeddings (Ruder et al, 2019), these data can be used to support explicit alignment objectives with either linear mappings (Wang et al, 2019(Wang et al, , 2020 or fine-tuning (Cao et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Do Explicit Alignments Robustly Improve Multilingual Encoders?

Dredze

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Multilingual BERT (Devlin et al., 2019, mBERT), XLM-RoBERTa (Conneau et al., 2019, XLMR) and other unsupervised multilingual encoders can effectively learn crosslingual representation. Explicit alignment objectives based on bitexts like Europarl or Mul-tiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bitexts are unavailable for many languages. In this paper, we propose a new contrastive alignment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sampled 1 million pair subset of the OPUS collection. Additionally, rather than report results on a single dataset with a single model run, we report the mean and standard derivation of multiple runs with different seeds, on four datasets and tasks. Our more extensive analysis finds that, while our new objective outperforms previous work, overall these methods do not improve performance with a more robust evaluation framework. Furthermore, the gains from using a better underlying model eclipse any benefits from alignment training. These negative results dictate more care in evaluating these methods and suggest limitations in applying explicit alignment objectives.

show abstract

“…Many papers discuss NMT, emphasizing zero shot neural machine [ 31 , 32 , 33 ] techniques. The authors of [ 34 ] note that NMT requires smaller data sizes, as small as a few thousand training sentences. In [ 35 ], an extensive survey for low resource NMT is introduced.…”

Section: Background and Literature Surveymentioning

confidence: 99%

Machine Translation Utilizing the Frequent-Item Set Concept

Mahmoud

Mengash

2021

Sensors

View full text Add to dashboard Cite

In this paper, we introduce new concepts in the machine translation paradigm. We treat the corpus as a database of frequent word sets. A translation request triggers association rules joining phrases present in the source language, and phrases present in the target language. It has to be noted that a sequential scan of the corpus for such phrases will increase the response time in an unexpected manner. We introduce the pre-processing of the bilingual corpus through proposing a data structure called Corpus-Trie (CT) that renders a bilingual parallel corpus in a compact data structure representing frequent data items sets. We also present algorithms which utilize the CT to respond to translation requests and explore novel techniques in exhaustive experiments. Experiments were performed on specific language pairs, although the proposed method is not restricted to any specific language. Moreover, the proposed Corpus-Trie can be extended from bilingual corpora to accommodate multi-language corpora. Experiments indicated that the response time of a translation request is logarithmic to the count of unrepeated phrases in the original bilingual corpus (and thus, the Corpus-Trie size). In practical situations, 5–20% of the log of the number of the nodes have to be visited. The experimental results indicate that the BLEU score for the proposed CT system increases with the size of the number of phrases in the CT, for both English-Arabic and English-French translations. The proposed CT system was demonstrated to be better than both Omega-T and Apertium in quality of translation from a corpus size exceeding 1,600,000 phrases for English-Arabic translation, and 300,000 phrases for English-French translation.

show abstract

Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation

Cited by 47 publications

References 17 publications

A Survey of Multilingual Neural Machine Translation

A Survey of Multilingual Neural Machine Translation

Do Explicit Alignments Robustly Improve Multilingual Encoders?

Machine Translation Utilizing the Frequent-Item Set Concept

Contact Info

Product

Resources

About