Self-attention in vision transformers performs perceptual grouping, not attention

Abstract

Recently, a considerable number of studies in computer vision involve deep neural architectures called vision transformers. Visual processing in these models incorporates computational models that are claimed to implement attention mechanisms. Despite an increasing body of work that attempts to understand the role of attention mechanisms in vision transformers, their effect is largely unknown. Here, we asked if the attention mechanisms in vision transformers exhibit similar effects as those known in human visual attention. To answer this question, we revisited the attention formulation in these models and found that despite the name, computationally, these models perform a special class of relaxation labeling with similarity grouping effects. Additionally, whereas modern experimental findings reveal that human visual attention involves both feed-forward and feedback mechanisms, the purely feed-forward architecture of vision transformers suggests that attention in these models cannot have the same effects as those known in humans. To quantify these observations, we evaluated grouping performance in a family of vision transformers. Our results suggest that self-attention modules group figures in the stimuli based on similarity of visual features such as color. Also, in a singleton detection experiment as an instance of salient object detection, we studied if these models exhibit similar effects as those of feed-forward visual salience mechanisms thought to be utilized in human visual attention. We found that generally, the transformer-based attention modules assign more salience either to distractors or the ground, the opposite of both human and computational salience. Together, our study suggests that the mechanisms in vision transformers perform perceptual organization based on feature similarity and not attention.

Can deep convolutional neural networks support relational reasoning in the same-different task?

“…In contrast, current object-centric representation learning methods do not make any attempt to capture this configural structure. Instead, their operation can be better described as perceptual segregation through clustering (Locatello et al, 2020;Mehrani and Tsotsos, 2023). Critically, there is substantial evidence that even the simple single-part abstract contour shapes used in the current study are represented by the human visual system as configurations of relations between segments of approximate constant curvature (Baker and Kellman, 2018;Baker, Garrigan and Kellman, 2021;.…”

“…Although a wide range of DNN architectures can be classified as object-centric, a common attribute across these models is the use of attention mechanisms that isolate individual objects. For most models, this is done through what has been termed affinity-based attention (Adeli, Ahn, Kriegeskorte and Zelinsky, 2023a) or perceptual grouping (Mehrani and Tsotsos, 2023). This process can be described as a bottom-up clustering of the visual input based on perceptual features such as color, texture and position (Mehrani and Tsotsos, 2023;Locatello et al, 2020).…”

“…For most models, this is done through what has been termed affinity-based attention (Adeli, Ahn, Kriegeskorte and Zelinsky, 2023a) or perceptual grouping (Mehrani and Tsotsos, 2023). This process can be described as a bottom-up clustering of the visual input based on perceptual features such as color, texture and position (Mehrani and Tsotsos, 2023;Locatello et al, 2020).…”

Printed circuit board solder joint quality inspection based on lightweight classification network

“…By splitting the convolution calculation process and integrating the SA method, it skillfully combines CNN and Transformer to expand feature representation capabilities. Subsequently, MobileVitv2 [39] proposed a linear time complexity SA method, which greatly reduced the resource occupation of the model. MobileVitv3 [40] improved the generalisation of the model by integrating context multi‐scale features.…”