Exploiting Source-side Monolingual Data in Neural Machine Translation

Zhang, Jiajun

doi:10.18653/v1/d16-1160

Cited by 256 publications

(222 citation statements)

References 29 publications

Supporting

Mentioning

219

Contrasting

Order By: Relevance

“…Different works have tackled the inclusion of monolingual data, either in source (Zhang and Zong, 2016b) and target language (Gulcehre et al, 2015(Gulcehre et al, , 2017.…”

Section: Related Workmentioning

confidence: 99%

Adapting Neural Machine Translation with Parallel Synthetic Data

Chinea-Rios¹,

Peris²,

Casacuberta³

2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

Recent works have shown that the usage of a synthetic parallel corpus can be effectively exploited by a neural machine translation system. In this paper, we propose a new method for adapting a general neural machine translation system to a specific task, by exploiting synthetic data.The method consists in selecting, from a large monolingual pool of sentences in the source language, those instances that are more related to a given test set. Next, this selection is automatically translated and the general neural machine translation system is fine-tuned with these data.For evaluating the adaptation method, we first conducted experiments in two controlled domains, with common and wellstudied corpora. Then, we evaluated our proposal on a real e-commerce task, yielding consistent improvements in terms of translation quality.

show abstract

“…Different works have tackled the inclusion of monolingual data, either in source (Zhang and Zong, 2016b) and target language (Gulcehre et al, 2015(Gulcehre et al, , 2017.…”

Section: Related Workmentioning

confidence: 99%

Adapting Neural Machine Translation with Parallel Synthetic Data

Chinea-Rios¹,

Peris²,

Casacuberta³

2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

show abstract

“…However, training with respect to the new loss is often computationally intensive and requires approximations. Alternatively, multi-task learning has been used to incorporate source-side (Zhang and Zong, 2016) and target-side (Domhan and Hieber, 2017) monolingual data. Another way of utilizing monolingual data in both source and target language is to warm start Seq2Seq training from pre-trained encoder and decoder networks (Ramachandran et al, 2017;Skorokhodov et al, 2018).…”

Section: Other Approachesmentioning

confidence: 99%

Simple Fusion: Return of the Language Model

Stahlberg¹,

Cross²,

Stoyanov³

2018

Proceedings of the Third Conference on Machine Translation: Research Papers

View full text Add to dashboard Cite

Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the training data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architecture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to utilize monolingual data such as backtranslation, shallow fusion, and cold fusion.

show abstract

“…Our approach further relates to Zhang and Zong (2016), who investigate multi-task learning for sequenceto-sequence models by strengthening the encoder using source-side monolingual data. A shared encoder architecture is used to predict both, transla-tions of parallel source sentences and permutations of monolingual source sentences.…”

Section: Related Workmentioning

confidence: 99%

Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning

Domhan

Hieber

2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The performance of Neural Machine Translation (NMT) models relies heavily on the availability of sufficient amounts of parallel data, and an efficient and effective way of leveraging the vastly available amounts of monolingual data has yet to be found. We propose to modify the decoder in a neural sequence-to-sequence model to enable multi-task learning for two strongly related tasks: target-side language modeling and translation. The decoder predicts the next target word through two channels, a target-side language model on the lowest layer, and an attentional recurrent model which is conditioned on the source representation. This architecture allows joint training on both large amounts of monolingual and moderate amounts of bilingual data to improve NMT performance. Initial results in the news domain for three language pairs show moderate but consistent improvements over a baseline trained on bilingual data only.

show abstract

Exploiting Source-side Monolingual Data in Neural Machine Translation

Cited by 256 publications

References 29 publications

Adapting Neural Machine Translation with Parallel Synthetic Data

Adapting Neural Machine Translation with Parallel Synthetic Data

Simple Fusion: Return of the Language Model

Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning

Contact Info

Product

Resources

About