Generating Diverse Translation by Manipulating Multi-Head Attention

Abstract

Transformer model (Vaswani et al. 2017) has been widely used in machine translation tasks and obtained state-of-theart results. In this paper, we report an interesting phenomenon in its encoder-decoder multi-head attention: different attention heads of the final decoder layer align to different word translation candidates. We empirically verify this discovery and propose a method to generate diverse translations by manipulating heads. Furthermore, we make use of these diverse translations with the back-translation technique for better data augmentation. Experiment results show that our method generates diverse translations without a severe drop in translation quality. Experiments also show that back-translation with these diverse translations could bring a significant improvement in performance on translation tasks. An auxiliary experiment of conversation response generation task proves the effect of diversity as well.

Generating Diverse Translation from Model Distribution with Dropout

“…In our experiment, we set the number of hidden states is 5. • Head Sampling (Sun et al, 2019): it generate different translations by sampling different encoder-decoder attention heads according to their attention weight, and copying the samples to other heads in some conditions. Here, we set the parameter K = 3.…”

“…He et al (2018) and Shen et al (2019) introduced latent variables into the NMT model, thus the model can generate diverse outputs using different latent variables. Moreover, Sun et al (2019) proposed to combine the structural characteristics of Transformer and use the different weights between each head in the multi-head attention mechanism to obtain diverse results. In spite of improvement in balancing accuracy and diversity, these methods do not represent the diversity in the NMT model directly.…”

“…Li et al (2016);Vijayakumar et al (2016) adjust decoding algorithms, adding different kinds of diversity regularization terms to encourage generating diverse outputs He et al (2018)Shen et al (2019) utilize mixture of experts (MoE) method, using differentiated latent variables to control generation of translation Sun et al (2019). generates diverse translation by sampling heads in encoder-decoder attention module in Transformer model, since different heads may present different target-source alignment Shu et al (2019).…”