Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

Abstract

Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both the general-domain and in-domain. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, domain divergence, and model explosion.To address these three problems, we propose a method of "divide and conquer" which is based on the importance of neurons or parameters in the translation model. In our method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and indomain translation. Then we further train the pruned model supervised by the original unpruned model with the knowledge distillation method. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conduct experiments on different languages and domains and the results show that our method can achieve significant improvements compared with several strong baselines.

What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis

“…While some work has been done to interpret representations in the speech models [38,31,32,33,34,39], no prior work has been carried to do a neuronlevel analysis. Analyzing individual neurons facilitates a deeper understanding of the network [40,41,42,23,36,37,43] and entails many potential benefits such as manipulating system's output [44] while debasing the network w.r.t certain property (like gender or racial elements), model distillation and compression [45,46], domain adaptation [47], feature selection for downstream tasks [48] and guiding architectural search etc. To develop a better understanding of the learned representations, along with presence of bias and redundancy, we carry a layer and neuron-level analysis on the speech models.…”