Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

Abstract

Attention is a key component of Transformers, which have recently achieved considerable success in natural language processing. Hence, attention is being extensively studied to investigate various linguistic capabilities of Transformers, focusing on analyzing the parallels between attention weights and specific linguistic phenomena. This paper shows that attention weights alone are only one of the two factors that determine the output of attention and proposes a norm-based analysis that incorporates the second factor, the norm of the transformed input vectors. The findings of our norm-based analyses of BERT and a Transformer-based neural machine translation system include the following: (i) contrary to previous studies, BERT pays poor attention to special tokens, and (ii) reasonable word alignment can be extracted from attention mechanisms of Transformer. These findings provide insights into the inner workings of Transformers.

A Primer in BERTology: What We Know About How BERT Works

“…More recently, Kobayashi et al (2020) showed that the norms of attention-weighted input vectors, which yield a more intuitive interpretation of selfattention, reduce the attention to special tokens. However, even when the attention weights are normed, it is still not the case that most heads that do the ''heavy lifting'' are even potentially interpretable (Prasanna et al, 2020).…”

Effective Attention Sheds Light On Interpretability

“…As an alternative to analyzing attention weights, Kobayashi et al (2020) propose anayzing the norm of vectors produced by multiplying the outputs of the value matrix with the attention weights. Follow-ing the experimental setting of Clark et al (2019), i.e., by analyzing 992 sequences extracted from Wikipedia, their norm-based analysis also shows that the contributions of [SEP] and punctuations are actually small.…”

Attention Head Masking for Inference Time Content Selection in Abstractive Summarization

“…Among the work that is relevant to encoder-decoder attentions, Michel et al (2019) and Voita et al (2019) observe that only a small portion of heads is relevant for translation and encoder-decoder attentions tend to be more important than self-attentions. Meanwhile, word alignments for machine translation are induced from encoder-decoder attention weights (Li et al, 2019;Kobayashi et al, 2020). However, none of prior work employs attentions to improve generation quality.…”