UNETR: Transformers for 3D Medical Image Segmentation

Hatamizadeh, Ali; Yang, Dong; Roth, Holger R.; Xu, Daguang

doi:10.48550/arxiv.2103.10504

Cited by 65 publications

(90 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Transformer-based models have recently gained a lot of attraction in computer vision [14,42,25] and medical image analysis [11,16]. Chen et al [11] introduced a 2D U-Net architecture that benefits from a ViT in the bottleneck of the network.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, ViTs have achieved success in effective learning of pretext tasks for self-supervised pre-training in various applications [9,8,36]. In medical image analysis, UNETR [16] is the first methodology that utilizes a ViT as its encoder without relying on a CNN-based feature extractor. Other approaches [40,39] have attempted to leverage the power of ViTs as a stand-alone block in their architectures which otherwise consist of CNN-based components.…”

Section: Introductionmentioning

confidence: 99%

“…Other approaches [40,39] have attempted to leverage the power of ViTs as a stand-alone block in their architectures which otherwise consist of CNN-based components. However, UNETR has shown better performance in terms of both accuracy and efficiency in different medical image segmentation tasks [16].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Hatamizadeh¹,

Nath²,

Tang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular "U-shaped" network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited kernel size of convolution layers in FCNNs, their performance of modeling long-range information is sub-optimal, and this can lead to deficiencies in the segmentation of tumors with variable sizes. On the other hand, transformer models have demonstrated excellent capabilities in capturing such longrange information in multiple domains, including natural language processing and computer vision. Inspired by the success of vision transformers and their variants, we propose a novel segmentation model termed Swin UNEt TRansformers (Swin UNETR). Specifically, the task of 3D brain tumor semantic segmentation is reformulated as a sequence to sequence prediction problem wherein multi-modal input data is projected into a 1D sequence of embedding and used as an input to a hierarchical Swin transformer as the encoder. The swin transformer encoder extracts features at five different resolutions by utilizing shifted windows for computing self-attention and is connected to an FCNN-based decoder at each resolution via skip connections. We have participated in BraTS 2021 segmentation challenge, and our proposed model ranks among the top-performing approaches in the validation phase. Code: https://monai.io/research/swin-unetr

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Hatamizadeh¹,

Nath²,

Tang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Transformers in Medical Image Segmentation. A number of recent studies Cao et al, 2021;Xie et al, 2021b;Hatamizadeh et al, 2021;Valanarasu et al, 2021;Isensee et al, 2021;Zheng et al, 2020) (Cao et al, 2021) explored how to use a pure transformer for medical image analysis tasks. However, the results do not lead to better performance.…”

Section: Related Workmentioning

confidence: 99%

Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

You,

Zhao,

Liu

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CA-GANformer, a novel type of generative adversarial transformers, for medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CA-GANformer dramatically outperforms previous state-of-the-art transformerbased approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model's inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CA-GANformer a strong starting point for downstream medical im-

show abstract

“…Transformers are particularly suited for this unique application, since aneurysms can be extremely small relative to a patient's overall brain mass, and they need to be carefully identified by type and characterized in fine detail to determine the appropriate medical or surgical intervention. Although a few studies have been published that use Transformer Neural Network models to perform tasks such as brain tumor segmentation, retinal blood vessel segmentation, and polyp segmentation in colonoscopy images [19][20][21][22][23][24][25][26], no studies to our knowledge have employed a Transformer architecture to detect aneurysms. This study proposes the use of a novel Transformer Deep Learning Neural Network approach to identify intracranial aneurysms in 3D MRA scans.…”

Section: Introductionmentioning

confidence: 99%

Computer-Assisted Detection Of Intracranial Aneurysms Using A Transformer Deep Neural Network In 3D MR Angiography

Zhou¹,

Huang

2022

Preprint

View full text Add to dashboard Cite

Intracranial aneurysms, a brain condition caused by weakened blood vessels, result in abnormal bulging or dilation of the intracranial artery walls. Up to 5% of the global population will suffer from this disease, and early detection is critical. Due to their small size, aneurysms are often missed in initial patient image assessments. Further, the process of interpreting 3D medical images can be time-consuming, error-prone, and require trained radiologists. This study implemented a deep learning model using a Transformer Neural Network to enable the automated computer-assisted detection of aneurysms in 3D TOF-MRA images. This novel Transformer model demonstrated overall 91.9% patient-level sensitivity and 95.1% patient specificity in detecting intracranial aneurysms. Additionally, for aneurysms exceeding 5 mm in diameter, it reached patient sensitivity of 95.8% and patient specificity of 96.4%. To our knowledge, this is the first study to implement a Transformer Deep Neural Network for the detection of aneurysms from 3D volumetric TOF-MRA images. The successful results underscore the potential of Transformer models in accelerating diagnoses of cerebrovascular diseases while streamlining radiologists’ workflow. This detection method can be applicable for identifying aneurysms in other organs and diseases such as brain tumors.

show abstract

UNETR: Transformers for 3D Medical Image Segmentation

Cited by 65 publications

References 44 publications

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

Computer-Assisted Detection Of Intracranial Aneurysms Using A Transformer Deep Neural Network In 3D MR Angiography

Contact Info

Product

Resources

About