Token-level Adaptive Training for Neural Machine Translation

Gu, Shuhao; Zhang, Jinchao; Meng, Fandong; Feng, Yang; Xie, Wanying; Zhou, Jie; Yu, Dong

doi:10.18653/v1/2020.emnlp-main.76

Cited by 22 publications

(30 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use a strong baseline system in this work in order to make the evaluation convincing. Improvement of existing methods (Gu et al, 2020) (Koehn, 2004) by contrast to all other models (p<0.01). Zh-En task, respectively.…”

Section: Resultsmentioning

confidence: 94%

See 2 more Smart Citations

Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Xu¹,

Liu²,

Meng³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

Recently, token-level adaptive training has achieved promising improvement in machine translation, where the cross-entropy loss function is adjusted by assigning different training weights to different tokens, in order to alleviate the token imbalance problem. However, previous approaches only use static word frequency information in the target language without considering the source language, which is insufficient for bilingual tasks like machine translation. In this paper, we propose a novel bilingual mutual information (BMI) based adaptive objective, which measures the learning difficulty for each target token from the perspective of bilingualism, and assigns an adaptive weight accordingly to improve token-level adaptive training. This method assigns larger training weights to tokens with higher BMI, so that easy tokens are updated with coarse granularity while difficult tokens are updated with fine granularity. Experimental results on WMT14 English-to-German and WMT19 Chinese-to-English demonstrate the superiority of our approach compared with the Transformer baseline and previous token-level adaptive training approaches. Further analyses confirm that our method can improve the lexical diversity.

show abstract

Section: Resultsmentioning

confidence: 94%

“…Exponential (Gu et al, 2020). This method adds an additional training weights to lowfrequency target tokens:…”

Section: Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Xu¹,

Liu²,

Meng³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

show abstract

“…Wang and Sennrich (2020) links the exposure bias problem (Ranzato et al, 2016;Shao et al, 2018;Zhang et al, 2019) to the phenomenon of NMT tends to generate hallucinations under domain shift. Gu et al (2020) finds that the NMT tends to generate more high-frequency tokens and less low-frequency tokens than reference. Compared with them, this work mainly focuses on investigating the functions of the different modules and parameters in the NMT model during continual training.…”

Section: Related Workmentioning

confidence: 94%

Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

Feng

2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data which may have a different distribution, e.g. a different domain. Although many methods have been proposed to solve this problem, we cannot get to know what causes this phenomenon yet. Under the background of domain adaptation, we investigate the cause of catastrophic forgetting from the perspectives of modules and parameters (neurons). The investigation on the modules of the NMT model shows that some modules have tight relation with the general-domain knowledge while some other modules are more essential in the domain adaptation. And the investigation on the parameters shows that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general-domain. We conduct experiments across different language pairs and domains to ensure the validity and reliability of our findings.

show abstract

“…where l ( x n i , y n i , t) stands for the cross-entropy loss of the ith token of the example x n , y n of the tth training step. We ensure all weights to be larger than 1 to ensure the gradient norm during backpropagation (Gu et al, 2020).…”

Section: Data Weightmentioning

confidence: 99%

Self-Supervised Curriculum Learning for Spelling Error Correction

Gan¹,

Xu²,

Hongying³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Spelling Error Correction (SEC) that requires high-level language understanding is a challenging but useful task. Current SEC approaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) utilizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model's performance has been shown sensitive to the difficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are naturally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speakers are hard), and we expect that designing a curriculum correspondingly for model learning may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and propose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model training, which in return improves the CL measurement. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and establish a new state of the art (74.38% F1).

show abstract

Token-level Adaptive Training for Neural Machine Translation

Cited by 22 publications

References 30 publications

Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

Self-Supervised Curriculum Learning for Spelling Error Correction

Contact Info

Product

Resources

About