Understanding Data Augmentation in Neural Machine Translation: Two Perspectives towards Generalization

Li, Guanlin; Liu, Lemao; Huang, Guoping; Zhu, Conghui; Zhao, Tiejun

doi:10.18653/v1/d19-1570

Cited by 19 publications

(14 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A translation model is then trained using both the pseudo-parallel and original-parallel data. Li et al (2019) analyzed multiple data augmentation methods. In their experiments, they applied self-training and back-translation.…”

Section: Related Workmentioning

confidence: 99%

“…First, we back-translate the target-side of the parallel corpus (Li et al, 2019;Sennrich et al, 2016) to create pseudo data as additional training data. Note that we do not use external data in backtranslation, and the diversity of target sentences does not change.…”

Section: Data Augmentation By Sentence Concatenationmentioning

confidence: 99%

“…There-fore, various data augmentation methods for lowresource parallel corpora have been studied. For instance, the generation of pseudo data was proposed by back-translating the monolingual corpora or paraphrasing the parallel corpora as additional training data (Wang et al, 2018;Sennrich et al, 2016;Li et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

Kondo¹,

Hotate²,

Hirasawa³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Rese

View full text Add to dashboard Cite

Neural machine translation (NMT) has recently gained widespread attention because of its high translation accuracy. However, it shows poor performance in the translation of long sentences, which is a major issue in low-resource languages. It is assumed that this issue is caused by insufficient number of long sentences in the training data. Therefore, this study proposes a simple data augmentation method to handle long sentences. In this method, we use only the given parallel corpora as the training data and generate long sentences by concatenating two sentences. Based on the experimental results, we confirm improvements in long sentence translation by the proposed data augmentation method, despite its simplicity. Moreover, the translation quality is further improved by the proposed method, when combined with backtranslation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Data Augmentation By Sentence Concatenationmentioning

confidence: 99%

See 1 more Smart Citation

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

Kondo¹,

Hotate²,

Hirasawa³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Rese

View full text Add to dashboard Cite

show abstract

“…Margin (Bartlett et al, 2017) is a classic concept in support vector machine, measuring the geometric distance between the support vectors and the decision boundary. To apply margin for NMT models, we follow Li et al (2019) to compute wordwise margin, which is defined as the probability of the correctly predicted word minus the maximum probability of other word types. We compute the word-wise margin over the training set and report the averaged value.…”

Section: A3 Generalization Capabilitymentioning

confidence: 99%

Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation

Jiao

Wang

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Large-scale training datasets lie at the core of the recent success of neural machine translation (NMT) models. However, the complex patterns and potential noises in the large-scale data make training NMT models difficult. In this work, we explore to identify the inactive training examples which contribute less to the model performance, and show that the existence of inactive examples depends on the data distribution. We further introduce data rejuvenation to improve the training of NMT models on large-scale datasets by exploiting inactive examples. The proposed framework consists of three phases. First, we train an identification model on the original training data, and use it to distinguish inactive examples and active examples by their sentence-level output probabilities. Then, we train a rejuvenation model on the active examples, which is used to re-label the inactive examples with forwardtranslation. Finally, the rejuvenated examples and the active examples are combined to train the final NMT model. Experimental results onWMT14 English-German and English-French datasets show that the proposed data rejuvenation consistently and significantly improves performance for several strong NMT models. Extensive analyses reveal that our approach stabilizes and accelerates the training process of NMT models, resulting in final models with better generalization capability. 1 * Work was mainly done when Wenxiang Jiao and Shilin He were interning at Tencent AI Lab.

show abstract

“…Similar steps are performed for IWSLT17 Fr-En. For the Zh-En dataset, we mainly adopt the same data-split method as Nguyen, Daumé, and Boyd-Graber (2017) and Li et al (2019), except that we use both their 'Supervised training' and 'Bandit training' sets as the training set. We use the Stanford Chinese word segmenter (Chang, Galley, and Manning 2008) to segment Chinese sentences.…”

mentioning

confidence: 99%

Balancing Quality and Human Involvement: An Effective Approach to Interactive Neural Machine Translation

Zhao

Liu

Huang

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

Conventional interactive machine translation typically requires a human translator to validate every generated target word, even though most of them are correct in the advanced neural machine translation (NMT) scenario. Previous studies have exploited confidence approaches to address the intensive human involvement issue, which request human guidance only for a few number of words with low confidences. However, such approaches do not take the history of human involvement into account, and optimize the models only for the translation quality while ignoring the cost of human involvement. In response to these pitfalls, we propose a novel interactive NMT model, which explicitly accounts the history of human involvements and particularly is optimized towards two objectives corresponding to the translation quality and the cost of human involvement, respectively. Specifically, the model jointly predicts a target word and a decision on whether to request human guidance, which is based on both the partial translation and the history of human involvements. Since there is no explicit signals on the decisions of requesting human guidance in the bilingual corpus, we optimize the model with the reinforcement learning technique which enables our model to accurately predict when to request human guidance. Simulated and real experiments show that the proposed model can achieve higher translation quality with similar or less human involvement over the confidence-based baseline.

show abstract

Understanding Data Augmentation in Neural Machine Translation: Two Perspectives towards Generalization

Cited by 19 publications

References 21 publications

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation

Balancing Quality and Human Involvement: An Effective Approach to Interactive Neural Machine Translation

Contact Info

Product

Resources

About