Alleviating the Inequality of Attention Heads for Neural Machine Translation

Abstract

Recent studies show that the attention heads in Transformer are not equal Michel et al., 2019). We relate this phenomenon to the imbalance training of multihead attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.