Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting selfattention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (86.4 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The code and models will be made publicly available at https:// github.com/microsoft/Swin-Transformer.* Equal contribution. † Interns at MSRA. ‡ Contact person.

Training Vision Transformers with Only 2040 Images

“…Many ViT variants were proposed in recent months. Swin Transformer [21] applied the shifted window approach to compute self-attention matrix. Wang et al proposed PVT-based model (PVTv1 & v2) [34,35], which built a progressive shrinking pyramid and a spatial-reduction attention layer to generate multi-resolution feature maps.…”

“…In order to alleviate this problem, a lot of works try to introduce convolutions to ViTs [21,35,37,40]. These architectures enjoy the advantages of both paradigms, with attention layers modeling long-range dependencies while convolutions emphasizing the local properties of images.…”

CMC2R: Cross‐modal collaborative contextual representation for RGBT tracking

“…Beyond classification, PVT [16] introduces a pyramid structure in Transformer, demonstrating the potential of a pure transformer backbone compared to CNN counterparts in dense prediction tasks. After that, methods such as Swin [17], CvT [18] and Twins [19] enhance the local continuity of features and remove fixed size position embedding to improve the performance of Transformers in dense prediction tasks.…”

“…To reduce the complexity of this process for large resolutions, the sequence reduction process [17] is used, this process uses a reduction ratio R to reduce the length of the sequence of as follows:…”

Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

“…Inspired by the recent success of vision transformer networks (Zhang et al, 2019;Carion et al, 2020;Zeng et al, 2020;Dosovitskiy et al, 2020;Esser et al, 2021;Liu et al, 2021;Hudson & Zitnick, 2021;Touvron et al, 2020;, we make a step towards a more practical scenario in which we only assume access to pre-trained models on public computer vision datasets, and a relatively small medical dataset, which we can use the weights of the pre-trained models to achieve higher accuracy in the medical image analysis tasks. These settings are particularly appealing as (1) such models can be easily adopted on typical medical datasets;…”