Self-Paced Learning for Neural Machine Translation

Wan, Yu; Yang, Baosong; Wong, Derek F.; Zhou, Yikai; Chao, Lidia S.; Zhang, Haibo; Chen, Boxing

doi:10.18653/v1/2020.emnlp-main.80

Cited by 30 publications

(26 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By heightening the exposure of low-frequency tokens during training, these models can meliorate the neglect of low-frequency tokens and improve lexical diversity of the translations. However, simply promoting low-frequency tokens via loss re-weighting may potentially sacrifice the learning of high-frequency ones (Gu et al 2020;Wan et al 2020;Zhou et al 2020). Besides, our further investigation on these methods reveals that generating more unusual tokens comes at the unexpected expense of their prediction precision (Section 5.4).…”

Section: Introductionmentioning

confidence: 93%

Frequency-Aware Contrastive Learning for Neural Machine Translation

Zhang¹,

Ye²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Low-frequency word prediction remains a challenge in modern neural machine translation (NMT) systems. Recent adaptive training methods promote the output of infrequent words by emphasizing their weights in the overall training objectives. Despite the improved recall of low-frequency words, their prediction precision is unexpectedly hindered by the adaptive objectives. Inspired by the observation that lowfrequency words form a more compact embedding space, we tackle this challenge from a representation learning perspective. Specifically, we propose a frequency-aware tokenlevel contrastive learning method, in which the hidden state of each decoding step is pushed away from the counterparts of other target words, in a soft contrastive way based on the corresponding word frequencies. We conduct experiments on widely used NIST Chinese-English and WMT14 English-German translation tasks. Empirical results show that our proposed methods can not only significantly improve the translation quality but also enhance lexical diversity and optimize word representation space. Further investigation reveals that, comparing with related adaptive training strategies, the superiority of our method on low-frequency word prediction lies in the robustness of token-level recall across different frequencies without sacrificing precision.

show abstract

Section: Introductionmentioning

confidence: 93%

Frequency-Aware Contrastive Learning for Neural Machine Translation

Zhang¹,

Ye²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, (Cai et al, 2020) proposes an adaptive multi-curricula learning framework to train the dialogue model with easy-to-complex dataset based on various concepts of difficulty including the specificity and repetitiveness of the response, the relevance between the query and the response, etc. Also, Wan et al (2020) resolves this problem by introducing selfpaced learning (Kumar et al, 2010), which is a special kind of curriculum learning (Eppe et al, 2019). Wan et al (2020) measures the level of confidence on each training example, where an easy sample is actually one of high confidence by the current trained model.…”

Section: Introductionmentioning

confidence: 99%

“…Also, Wan et al (2020) resolves this problem by introducing selfpaced learning (Kumar et al, 2010), which is a special kind of curriculum learning (Eppe et al, 2019). Wan et al (2020) measures the level of confidence on each training example, where an easy sample is actually one of high confidence by the current trained model. Both curriculum learning and self-paced learning suggest that samples should be selected in a meaningful order for training.…”

Section: Introductionmentioning

confidence: 99%

Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation

Zhu¹,

Chen²,

Wu³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Curriculum learning, a machine training strategy that feeds training instances to the model from easy to hard, has been proven to facilitate the dialogue generation task. Meanwhile, knowledge distillation, a knowledge transformation methodology among teachers and students networks can yield significant performance boost for student models. Hence, in this paper, we introduce a combination of curriculum learning and knowledge distillation for efficient dialogue generation models, where curriculum learning can help knowledge distillation from data and model aspects. To start with, from the data aspect, we cluster the training cases according to their complexity, which is calculated by various types of features such as sentence length and coherence between dialog pairs. Furthermore, we employ an adversarial training strategy to identify the complexity of cases from model level. The intuition is that, if a discriminator can tell the generated response is from the teacher or the student, then the case is difficult that the student model has not adapted to yet. Finally, we use selfpaced learning, which is an extension to curriculum learning to assign weights for distillation. In conclusion, we arrange a hierarchical curriculum based on the above two aspects for the student model under the guidance from the teacher model. Experimental results demonstrate that our methods achieve improvements compared with competitive baselines.

show abstract

“…Neural machine translation (NMT) employs an endto-end framework (Sutskever et al, 2014) and has advanced promising results on various sentencelevel translation tasks (Bahdanau et al, 2015;Gehring et al, 2017;Vaswani et al, 2017;Wan et al, 2020). However, most of NMT models handle sentences independently, regardless of the linguistic context that may appear outside the current sentence (Tiedemann and Scherrer, 2017a).…”

Section: Introductionmentioning

confidence: 99%

Multi-Hop Transformer for Document-Level Machine Translation

Zhang¹,

Zhang

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Document-level neural machine translation (NMT) has proven to be of profound value for its effectiveness on capturing contextual information. Nevertheless, existing approaches 1) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process; and 2) feed ground-truth target contexts as extra inputs at the training time, thus facing the problem of exposure bias. We approach these problems with an inspiration from human behavior -human translators ordinarily emerge a translation draft in their mind and progressively revise it according to the reasoning in discourse. To this end, we propose a novel Multi-Hop Transformer (MHT) which offers NMT abilities to explicitly model the human-like draft-editing and reasoning process. Specifically, our model serves the sentence-level translation as a draft and properly refines its representations by attending to multiple antecedent sentences iteratively. Experiments on four widely used document translation tasks demonstrate that our method can significantly improve documentlevel translation performance and can tackle discourse phenomena, such as coreference error and the problem of polysemy.

show abstract

Self-Paced Learning for Neural Machine Translation

Cited by 30 publications

References 25 publications

Frequency-Aware Contrastive Learning for Neural Machine Translation

Frequency-Aware Contrastive Learning for Neural Machine Translation

Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation

Multi-Hop Transformer for Document-Level Machine Translation

Contact Info

Product

Resources

About