Shunted Self-Attention via Multi-Scale Token Aggregation

Ren, Sucheng; Zhou, Daohong; He, Shengfeng; Feng, Jiashi; Wang, Xinchao

doi:10.1109/cvpr52688.2022.01058

Cited by 201 publications

(60 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, we introduce a local context enhancement term LCE((V) as in [37]. Function LCE(•) is parametrized with a depth-wise convolution, and we set the kernel size to 5.…”

Section: Bi-level Routing Attention (Bra)mentioning

confidence: 99%

BiFormer: Vision Transformer with Bi-Level Routing Attention

Li¹,

Wang²,

Ke³

et al. 2023

Preprint

View full text Add to dashboard Cite

As the core building block of vision transformers, attention is a powerful tool to capture long-range dependency. However, such power comes at a cost: it incurs a huge computation burden and heavy memory footprint as pairwise token interaction across all spatial locations is computed. A series of works attempt to alleviate this problem by introducing handcrafted and content-agnostic sparsity into attention, such as restricting the attention operation to be inside local windows, axial stripes, or dilated windows. In contrast to these approaches, we propose a novel dynamic sparse attention via bi-level routing to enable a more flexible allocation of computations with content awareness. Specifically, for a query, irrelevant key-value pairs are first filtered out at a coarse region level, and then fine-grained token-to-token attention is applied in the union of remaining candidate regions (i.e., routed regions). We provide a simple yet effective implementation of the proposed bilevel routing attention, which utilizes the sparsity to save both computation and memory while involving only GPUfriendly dense matrix multiplications. Built with the proposed bi-level routing attention, a new general vision transformer, named BiFormer, is then presented. As BiFormer attends to a small subset of relevant tokens in a query adaptive manner without distraction from other irrelevant ones, it enjoys both good performance and high computational efficiency, especially in dense prediction tasks. Empirical results across several computer vision tasks such as image classification, object detection, and semantic segmentation verify the effectiveness of our design. Code is available at https://github.com/rayleizhu/BiFormer.

show abstract

“…Here, we introduce a local context enhancement term LCE((V) as in [37]. Function LCE(•) is parametrized with a depth-wise convolution, and we set the kernel size to 5.…”

Section: Bi-level Routing Attention (Bra)mentioning

confidence: 99%

BiFormer: Vision Transformer with Bi-Level Routing Attention

Li¹,

Wang²,

Ke³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…As a common sense in Transformer, we propose three variants for SCAT: tiny (t), small (s) and base (b), in order to make a close model size (number of parameters) as in Shunted Transformer 7 . Note that all variants of SCAT have the same embedding dimension as and hidden dimension for the lightweight decoder.…”

Section: Model Configurationsmentioning

confidence: 99%

“…We set the batch size for training as 32 and train 160K iterations for all models. Focal loss 15 is utilized for the experiments as follows: (7) where denotes the ground truth semantic segmentation map, and denotes the focal loss function 15 with and as the predicted probability that belong to the true class. is a trade-off hyper-parameter for balancing the losses of the coarse segmentation map and the refined prediction .…”

Section: Implementation Detailsmentioning

confidence: 99%

A transformer-based semantic segmentation model for street fashion images

PENG

Kameyama

2023

International Workshop on Advanced Imaging Technology (IWAIT) 2023

View full text Add to dashboard Cite

Semantic segmentation is a pixel-level classification problem in computer vision, in which pixels of the same class are grouped into a single category in order to interpret pictures at the pixel level. In this field, semantic segmentation of street fashion images is a challenging task since the clothing items would appear with wide variations in fabrics, layering, occlusion and viewpoint. To help better understanding the street fashion images, we propose a lightweight Semantic Context Aware Transformer (SCAT) to be applied to the semantic segmentation task for street fashion images, which integrates semantic context into the encoding, and models the relationship between multi-level outputs from transformer layers. Extensive experiments and comparisons show that the proposal achieves the state-of-the-art results on ModaNet dataset with relatively small model size, with over 1.1 point improvement compared to Shunted Transformer, and even surpasses other CNNs and Transformers with a large margin of over 2 point in mIoU.

show abstract

“…Visual attention has been widely used in deep learning and achieves remarkable advances [59,69]. It has been exploited in computer vision tasks such as image recognition [9,10,33,48,61,71] and object detection among others [6,16,35,66,67]. CAM [77] provides the attention visualization of feature maps for model interpretable analysis.…”

Section: Related Workmentioning

confidence: 99%

Attention Diversification for Domain Generalization

Meng

Chen

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have demonstrated gratifying results at learning discriminative features. However, when applied to unseen domains, state-of-the-art models are usually prone to errors due to domain shift. After investigating this issue from the perspective of shortcut learning, we find the devils lie in the fact that models trained on different domains merely bias to different domain-specific features yet overlook diverse task-related features. Under this guidance, a novel Attention Diversification framework is proposed, in which Intra-Model and Inter-Model Attention Diversification Regularization are collaborated to reassign appropriate attention to diverse task-related features. Briefly, Intra-Model Attention Diversification Regularization is equipped on the high-level feature maps to achieve in-channel discrimination and cross-channel diversification via forcing different channels to pay their most salient attention to different spatial locations. Besides, Inter-Model Attention Diversification Regularization is proposed to further provide task-related attention diversification and domain-related attention suppression, which is a paradigm of "simulate, divide and assemble": simulate domain shift via exploiting multiple domain-specific models, divide attention maps into task-related and domain-related groups, and assemble them within each group respectively to execute regularization. Extensive experiments and analyses are conducted on various benchmarks to demonstrate that our method achieves state-of-the-art performance over other competing methods. Code is available at https: //github.com/hikvision-research/DomainGeneralization.

show abstract

Shunted Self-Attention via Multi-Scale Token Aggregation

Cited by 201 publications

References 14 publications

BiFormer: Vision Transformer with Bi-Level Routing Attention

BiFormer: Vision Transformer with Bi-Level Routing Attention

A transformer-based semantic segmentation model for street fashion images

Attention Diversification for Domain Generalization

Contact Info

Product

Resources

About