How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

Abstract

The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find that this mechanism, while powerful and elegant, is not as important as typically thought for pretrained language models. We introduce PAPA, 1 a new probing method that replaces the input-dependent attention matrices with constant ones-the average attention weights over multiple inputs. We use PAPA to analyze several established pretrained Transformers on six downstream tasks. We find that without any input-dependent attention, all models achieve competitive performance-an average relative drop of only 8% from the probing baseline. Further, little or no performance drop is observed when replacing half of the input-dependent attention matrices with constant (input-independent) ones. Interestingly, we show that better-performing models lose more from applying our method than weaker models, suggesting that the utilization of the input-dependent attention mechanism might be a factor in their success. Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.

PMET: Precise Model Editing in a Transformer

“…We then observe that the knowledge contained within MHSA undergoes more frequent changes compared to that within FFN. Combining previous findings of MHSA (Geva et al 2023;Wang et al 2022;Hassid et al 2022) with our observation, we believe that MHSA works as a knowledge extractor and stores certain general knowledge extraction patterns. This suggests the potential for supplementary optimization of TC hidden states of the MHSA to expand the function space, without necessitating updates to its weights.…”

“…We attribute this observation to the fact that the MHSA continuously extracts various types of knowledge, while the FFN primarily extracts its own knowledge (Geva et al 2021;Meng et al 2022a). Furthermore, considering previous findings regarding the extraction of attributes from the MHSA with observed redundancies (Geva et al 2023;Wang et al 2022;Hassid et al 2022), we believe that the MHSA works as a knowledge extractor and stores certain general knowledge extraction patterns. Thus we suggest that when introducing new knowledge, there is no need to update the MHSA weights.…”

“…Post-hoc explanation is a broad field, and our focus is on understanding the roles of the two components, MHSA and FFN, in Transformers (Kovaleva et al 2019;Wang et al 2022;Hassid et al 2022;Geva et al 2022;Kobayashi et al 2023;Hao et al 2021;Geva et al 2023). Currently, it is widely believed that FFN serves as the main carrier for storing factual knowledge (Geva et al 2021;Meng et al 2022a,b), and each layer of FFN contributes to knowledge recall (Geva et al 2022(Geva et al , 2023.…”

“…MHSA is primarily responsible for capturing the degree of association between different tokens, focusing on interactions between content (Hao et al 2021;Kobayashi et al 2023), and extracting attributes of subjects (Geva et al 2023). Other studies have shown that MHSA contains different levels of redundant information (Wang et al 2022;Hassid et al 2022). These findings imply that MHSA may store certain general patterns used for knowledge extraction.…”

Efficient Information Modulation Network for Image Super-Resolution

“…Recently, the Transformer has attracted significant interest in the computer vision community, thanks to its powerful representation capabilities. However, several works have found that the excellence of Transformer comes from the macro-level framework and advanced components rather than its self-attention (SA) mechanism to some extent [38,36,12]. Surprisingly, comparable results on multiple mainstream computer vision tasks can still be obtained by replacing SA with spatial pooling [38], spatial shifting [36], spatial MLP [33,32,34,13,23], fourier transform [31,17] and constant matrix [12], all of which have spatial information encoding capability sim- Following this motivation, our mainly purpose is to design a new spatial information encoder for encoding spatial features efficiently by introducing large kernel convolution operation and convolutional modulation technology to realize long-range correlations and selfadaptive behavior like SA, within the powerful Transformer macro framework.…”