Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation

Abstract

This work presents our ongoing research of unsupervised pretraining in neural machine translation (NMT). In our method, we initialize the weights of the encoder and decoder with two language models that are trained with monolingual data and then fine-tune the model on parallel data using Elastic Weight Consolidation (EWC) to avoid forgetting of the original language modeling tasks. We compare the regularization by EWC with the previous work that focuses on regularization by language modeling objectives. The positive result is that using EWC with the decoder achieves BLEU scores similar to the previous work. However, the model converges 2-3 times faster and does not require the original unlabeled training data during the finetuning stage. In contrast, the regularization using EWC is less effective if the original and new tasks are not closely related. We show that initializing the bidirectional NMT encoder with a left-toright language model and forcing the model to remember the original left-to-right language modeling task limits the learning capacity of the encoder for the whole bidirectional context.

Continual Lifelong Learning in Natural Language Processing: A Survey

“…A stream of research focused on techniques more traditionally associated with CL has also been present. In the works of Miceli Barone et al (2017); ; Variš and Bojar (2019); Thompson et al (2019) regularization approaches (e.g. EWC) were leveraged.…”

More Parameters? No Thanks!

“…Adapting multilingual model to a new language pair and domain adaptation: Prior works on adaptation (Neubig and Hu, 2018;Variš and Bojar, 2019;Stickland et al, 2020;Escolano et al, 2020;Akella et al, 2020;Zhang et al, 2020) aims at improving language specific performance by either fine-tuning the same MNMT model or adding language specific modules. While being effective, these methods either lose their multilinguality or introduce additional parameters.…”

Monolingual Adapters for Zero-Shot Neural Machine Translation

“…Adaptation to a new language pair may be addressed by training a multilingual model then fine-tuning it with parallel data in the language pair of interest (Neubig and Hu, 2018;Variš and Bojar, 2019;Stickland et al, 2020). Escolano et al (2020) propose plug-and-play encoders and decoders per language, which take advantage of a single representation in each language but at the cost of larger model sizes.…”