Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

Abstract

Recently, Transformers have shown promising performance in various vision tasks. To reduce the quadratic computation complexity caused by the global self-attention, various methods constrain the range of attention within a local region to improve its efficiency. Consequently, their receptive fields in a single attention layer are not large enough, resulting in insufficient context modeling. To address this issue, we propose a Pale-Shaped self-Attention (PS-Attention), which performs self-attention within a pale-shaped region. Compared to the global self-attention, PS-Attention can reduce the computation and memory costs significantly. Meanwhile, it can capture richer contextual information under the similar computation complexity with previous local self-attention mechanisms. Based on the PS-Attention, we develop a general Vision Transformer backbone with a hierarchical architecture, named Pale Transformer, which achieves 83.4%, 84.3%, and 84.9% Top-1 accuracy with the model size of 22M, 48M, and 85M respectively for 224x224 ImageNet-1K classification, outperforming the previous Vision Transformer backbones. For downstream tasks, our Pale Transformer backbone performs better than the recent state-of-the-art CSWin Transformer by a large margin on ADE20K semantic segmentation and COCO object detection & instance segmentation. The code will be released on https://github.com/BR-IDL/PaddleViT.

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

“…Our models are highlighted in gray. Although some recent work [33,37,55] with hybrid architectures or more careful designs on model width and depth achieve better performance than HorNet on ImageNet-1K, we think our models will also benefit from these techniques and achieve better performance. Our models also generalize well to a larger image resolution, larger model sizes and more training data.…”

Transformer-Based GAN for New Hairstyle Generative Networks

“…Srinivas et al [27] proposed BoTNet, a transformer model incorporating the backbone architecture of selfattention and applied it to multiple computer vision tasks including image classification, object detection, and instance segmentation. Wu et al [28] proposed Pale Transformer with pale self-attention (PS-Attention) that performs self-attention within pale regions. Compared with global self-attention, PS-Attention can significantly reduce computational and memory costs while capturing richer contextual information.…”

Research on Intelligent Management of Laboratory Information Technology

“…Some other methods try to alleviate the computations of self-attention by reducing the number of tokens. One way is to enforce the computation of self-attention to be conducted in a predefined local region, for instance, Swin Transformer [23], Pale Transformer [33], HaloNet [30], and CSWin Transformer [6]. These methods are based on the assumption that image patches located far from each other are not semantically relevant, but this only partially holds true.…”

“…Accordingly, the recent efforts were devoted to the following trials: (1) Enforce the self-attention to be confined in a neighborhood around each token such that fewer tokens will be involved in updating each token. The methods falling in this category include Swin Transformer [23], Pale Transformer [33], HaloNet [30], and CSWin Transformer [6]. These methods are based on such an assumption that tokens spatially far away are not semantically correlated, but this does not always hold true.…”