Iterative Dual Domain Adaptation for Neural Machine Translation

Abstract

Previous studies on the domain adaptation for neural machine translation (NMT) mainly focus on the one-pass transferring out-ofdomain translation knowledge to in-domain NMT model. In this paper, we argue that such a strategy fails to fully extract the domainshared translation knowledge, and repeatedly utilizing corpora of different domains can lead to better distillation of domain-shared translation knowledge. To this end, we propose an iterative dual domain adaptation framework for NMT. Specifically, we first pretrain in-domain and out-of-domain NMT models using their own training corpora respectively, and then iteratively perform bidirectional translation knowledge transfer (from indomain to out-of-domain and then vice versa) based on knowledge distillation until the indomain NMT model convergences. Furthermore, we extend the proposed framework to the scenario of multiple out-of-domain training corpora, where the above-mentioned transfer is performed sequentially between the indomain and each out-of-domain NMT models in the ascending order of their domain similarities. Empirical results on Chinese-English and English-German translation tasks demonstrate the effectiveness of our framework.

Proceedings of the Fourth Workshop on Neural Generation and Translation

“…Ruder et al (2017) introduced this idea as "Knowledge Adaptation," using multi-layer perceptrons to provide sentiment analysis labels for unlabeled indomain data. Similar work includes Iterative Dual Domain Adaptation (Zeng et al, 2019) and Domain Transformation Networks independently showed that knowledge distillation could be used to compress pre-trained models without affecting downstream tasks. Tang et al (2019) showed that task-specific information could be distilled from a large Transformer into a much smaller Bi-directional RNN.…”

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

“…Britz et al (2017) adds a discriminator to extract common features across domains. There are also some work (Zeng et al, 2018(Zeng et al, , 2019Gu et al, 2019) that adds domain-specific modules to the model to preserve the domain-specific features. Currey et al (2020) distills multiple expert models into a single student model.…”

WIND: Weighting Instances Differentially for Model-Agnostic Domain Adaptation

“…For machine translation task, similar to the settings of Luong and Manning (2015); Wang et al (2017); Zeng et al (2019), we use the IWSLT 2016 English (EN) to German (DE) corpus (Cettolo et al, 2016) as the in-domain data. This corpus contains about 202K sentences from TED talks.…”

“…• IDDA This indicates Iterative Dual Domain Adaptation methods proposed by Zeng et al (2019), which iteratively performs bidirectional translation knowledge transfer using knowledge distillation between in-domain and out-of-domain. Note that this method focuses on the performance of both domains but in this paper we only focus on in-domain performance.…”