Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Tang, Yucheng; Yang, Dong; Li, Wenqi; Roth, Holger R.; Landman, Bennett A.; Xu, Daguang; Nath, Vishwesh; Hatamizadeh, Ali

doi:10.48550/arxiv.2111.14791

Cited by 19 publications

(26 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In another work, Tang et al [163] introduce Swin UNETR, a novel self-supervised learning framework with proxy tasks to pre-train Transformer encoder on 5,050 images of CT dataset. They validate the effectiveness of pre-training by finetuning the Transformer encoder with a CNN-based decoder on the downstream task of MSD and BTCV segmentation datasets.…”

Section: Hybrid Architecturesmentioning

confidence: 99%

“…TransUNet [96], CoTr* [112] (small CNN encoder compared to CoTr), CoTr [112], UNETR [35], and Swin UNETR [163]. Note: Avg: Average results (over 12 organs), AG: left and right adrenal glands, Pan: pancreas, Sto: stomach, Spl: spleen, Liv: liver, Gall: gallbladder.…”

Section: Multi-scale Architecturesmentioning

confidence: 99%

“…In short, although recent ViT-based data-efficient approaches like DeiT [389], Token-to-Token [290], transformer in transformer [216], etc., report encouraging results in the generic vision applications, the task of learning these transformer models tailored to domain-specific medical imaging applications in a data-efficient manner is challenging. Recently, Tang et al [163] has made an attempt to handle this issue by investigating the effectiveness of self-supervised learning as a pre-training strategy on domain-specific medical images. Specifically, they propose 3D transformer-based hierarchical encoder, Swin UNETR, and after pre-training on 5,050 CT images, demonstrates its effectiveness via fine-tuning on the downstream task of medical image segmentation.…”

Section: Pre-trainingmentioning

confidence: 99%

See 2 more Smart Citations

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Hybrid Architecturesmentioning

confidence: 99%

Section: Multi-scale Architecturesmentioning

confidence: 99%

Section: Pre-trainingmentioning

confidence: 99%

See 1 more Smart Citation

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, self-attention module in ViT-based models allows for modeling long-range information by pairwise interaction between token embeddings and hence leading to more effective local and global contextual representations [34]. In addition, ViTs have achieved success in effective learning of pretext tasks for self-supervised pre-training in various applications [9,8,36]. In medical image analysis, UNETR [16] is the first methodology that utilizes a ViT as its encoder without relying on a CNN-based feature extractor.…”

Section: Introductionmentioning

confidence: 99%

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Hatamizadeh¹,

Nath²,

Tang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular "U-shaped" network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited kernel size of convolution layers in FCNNs, their performance of modeling long-range information is sub-optimal, and this can lead to deficiencies in the segmentation of tumors with variable sizes. On the other hand, transformer models have demonstrated excellent capabilities in capturing such longrange information in multiple domains, including natural language processing and computer vision. Inspired by the success of vision transformers and their variants, we propose a novel segmentation model termed Swin UNEt TRansformers (Swin UNETR). Specifically, the task of 3D brain tumor semantic segmentation is reformulated as a sequence to sequence prediction problem wherein multi-modal input data is projected into a 1D sequence of embedding and used as an input to a hierarchical Swin transformer as the encoder. The swin transformer encoder extracts features at five different resolutions by utilizing shifted windows for computing self-attention and is connected to an FCNN-based decoder at each resolution via skip connections. We have participated in BraTS 2021 segmentation challenge, and our proposed model ranks among the top-performing approaches in the validation phase. Code: https://monai.io/research/swin-unetr

show abstract

“…Hierarchical models [5,20,24] are received significant interest in medical image analysis due to their advantages of modeling heterogeneous high-resolution radiography images. Recent works on vision transformers [8,18] show superior performance on visual representations compared to state-of-the-art convolution-based networks [12].…”

Section: Introductionmentioning

confidence: 99%

Characterizing Renal Structures with 3D Block Aggregate Transformers

Yu¹,

Tang²,

Zhou³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Efficiently quantifying renal structures can provide distinct spatial context and facilitate biomarker discovery for kidney morphology. However, the development and evaluation of transformer model to segment the renal cortex, medulla, and collecting system remains challenging due to data inefficiency. Inspired by the hierarchical structures in vision transformer, we propose a novel method using 3D block aggregation transformer for segmenting kidney components on contrast-enhanced CT scans. We construct the first cohort of renal substructures segmentation dataset with 116 subjects under institutional review board (IRB) approval. Our method yields the state-of-the-art performance (Dice of 0.8467) against the baseline approach of 0.8308 with the data-efficient design. The Pearson R achieves 0.9891 between the proposed method and manual standards, and indicates the strong correlation and reproducibility for volumetric analysis. We extend the proposed method to the public KiTS dataset, the method leads to improved accuracy compared to transformer-based approaches. We show that the 3D block aggregation transformer can achieve local communication between sequence representations without modifying self-attention, and it can serve as an accurate and efficient quantification tool for characterizing renal structures.

show abstract

Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Cited by 19 publications

References 34 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Characterizing Renal Structures with 3D Block Aggregate Transformers

Contact Info

Product

Resources

About