Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

Abstract

Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data which may have a different distribution, e.g. a different domain. Although many methods have been proposed to solve this problem, we cannot get to know what causes this phenomenon yet. Under the background of domain adaptation, we investigate the cause of catastrophic forgetting from the perspectives of modules and parameters (neurons). The investigation on the modules of the NMT model shows that some modules have tight relation with the general-domain knowledge while some other modules are more essential in the domain adaptation. And the investigation on the parameters shows that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general-domain. We conduct experiments across different language pairs and domains to ensure the validity and reliability of our findings.

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

“…Given the above, we aim to propose a method of domain adaptation that can not only deal with large domain divergence during domain transferring but also keep a stable model size during domain increase. Inspired by some analysis work on NMT (Bau et al, 2019;Voita et al, 2019;Gu and Feng, 2020), we find that only some important parameters in a well-trained NMT model are responsible for generating the translation and unimportant parameters can be erased without affecting the translation quality too much. According to these findings, we can preserve important parameters for the general-domain, while turning unimportant parameters for the in-domain.…”

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

“…Given the above, we propose a method of domain adaptation that can not only deal with large domain divergence during domain transferring but also keep a stable model size even with multiple domains. Inspired by the analysis work on NMT (Bau et al, 2019;Voita et al, 2019;Gu and Feng, 2020), we find that only some important parameters in a well-trained NMT model play an important role when generating the translation and unimportant parameters can be erased without affecting the translation quality too much. According to these findings, we can preserve important parameters for generaldomain translation, while tuning unimportant parameters for in-domain translation.…”

Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

“…Catastrophic forgetting refers to the loss of previously acquired knowledge in the model during transfer to a new task. To the best of our knowledge, catastrophic forgetting in MT models has only been studied within the context of inter-domain adaptation (Thompson et al, 2019;Gu and Feng, 2020), and not inter-lingual adaptation.…”