Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Tang, Yucheng; Yang, Dong; Li, Wenqi; Roth, Holger R.; Landman, Bennett A.; Xu, Daguang; Nath, Vishwesh; Hatamizadeh, Ali

doi:10.1109/cvpr52688.2022.02007

Cited by 377 publications

(135 citation statements)

References 25 publications

Supporting

Mentioning

131

Contrasting

Unclassified

Order By: Relevance

“…In this work, we explore the use of three neural networks for fetal fat segmentation. In particular, we compare the state-of-the-art models Residual 3D UNet, nn-UNet [14] and SWIN-UNetR [28]. In previous works [10,17], variants of 3D-UNet were used to automate fat segmentation.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Fetal Fat Quantification from MRI

Avisdris

Rabinowich

Daniel

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Normal fetal adipose tissue (AT) development is essential for perinatal well-being. AT, or simply fat, stores energy in the form of lipids. Malnourishment may result in excessive or depleted adiposity. Although previous studies showed a correlation between the amount of AT and perinatal outcome, prenatal assessment of AT is limited by lacking quantitative methods. Using magnetic resonance imaging (MRI), 3D fat-and water-only images of the entire fetus can be obtained from twopoint Dixon images to enable AT lipid quantification. This paper is the first to present a methodology for developing a deep learning (DL) based method for fetal fat segmentation based on Dixon MRI. It optimizes radiologists' manual fetal fat delineation time to produce annotated training dataset. It consists of two steps: 1) model-based semi-automatic fetal fat segmentations, reviewed and corrected by a radiologist; 2) automatic fetal fat segmentation using DL networks trained on the resulting annotated dataset. Segmentation of 51 fetuses was performed with the semi-automatic method. Three DL networks were trained. We show a significant improvement in segmentation times (3:38 hours → < 1 hour) and observer variability (Dice of 0.738 → 0.906) compared to manual segmentation. Automatic segmentation of 24 test cases with the 3D Residual U-Net, nn-UNet and SWIN-UNetR transformer networks yields a mean Dice score of 0.863, 0.787 and 0.856, respectively. These results are better than the manual observer variability, and comparable to automatic adult and pediatric fat segmentation. A radiologist reviewed and corrected six new independent cases segmented using the best performing network (3D Residual U-Net), resulting in a Dice score of 0.961 and a significantly reduced correction time of 15:20 minutes. Using these novel segmentation methods and short MRI acquisition time, whole body subcutaneous lipids can be quantified for individual fetuses in the clinic and large-cohort research.

show abstract

Section: Discussionmentioning

confidence: 99%

“…We evaluate three models: (1) Residual 3D U-Net [16]; (2) nn-UNet [14], and; (3) SWIN-UNetR [28]. The 3D U-Net and nn-UNet are fully convolutional (FCN) encoder-decoder networks, where the decoder network is connected to the encoder network through skip connections.…”

Section: Automatic Fetal Fat Segmentationmentioning

confidence: 99%

Automatic Fetal Fat Quantification from MRI

Avisdris

Rabinowich

Daniel

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Since labeling all samples in a dataset is an expensive and time-consuming process, we propose using a self-supervised approach for learning useful gait features from the input data. The self-supervised approach has emerged in recent years [ 17 , 18 , 19 , 20 ] and has been successfully applied to a number of problems [ 21 , 22 ]. The main goal of self-supervised learning is to learn useful data representations from the unlabeled data by creating a pretext task.…”

Section: Introductionmentioning

confidence: 99%

Gait Recognition with Self-Supervised Learning of Gait Features Based on Vision Transformers

Pinčić

Sušanj

Lenac

2022

Sensors

View full text Add to dashboard Cite

Gait is a unique biometric trait with several useful properties. It can be recognized remotely and without the cooperation of the individual, with low-resolution cameras, and it is difficult to obscure. Therefore, it is suitable for crime investigation, surveillance, and access control. Existing approaches for gait recognition generally belong to the supervised learning domain, where all samples in the dataset are annotated. In the real world, annotation is often expensive and time-consuming. Moreover, convolutional neural networks (CNNs) have dominated the field of gait recognition for many years and have been extensively researched, while other recent methods such as vision transformer (ViT) remain unexplored. In this manuscript, we propose a self-supervised learning (SSL) approach for pretraining the feature extractor using the DINO model to automatically learn useful gait features with the vision transformer architecture. The feature extractor is then used for extracting gait features on which the fully connected neural network classifier is trained using the supervised approach. Experiments on CASIA-B and OU-MVLP gait datasets show the effectiveness of the proposed approach.

show abstract

“…If these methods are applied to 3D medical images, they may fail to model the prior of a brain because they treat all recovered patches equally. Some recent work [24] pre-trains transformers for medical images but it neglects the symmetry of brain structures and the different importance of brain regions. Motivated by the above observations, we consider a novel transformer-based SSL framework for brain MRI segmentation.…”

Section: Introductionmentioning

confidence: 99%

Attentive Symmetric Autoencoder for Brain MRI Segmentation

Huang¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised learning methods based on image patch reconstruction have witnessed great success in training auto-encoders, whose pre-trained weights can be transferred to fine-tune other downstream tasks of image understanding. However, existing methods seldom study the various importance of reconstructed patches and the symmetry of anatomical structures, when they are applied to 3D medical images. In this paper we propose a novel Attentive Symmetric Auto-encoder (ASA) based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. We conjecture that forcing the auto-encoder to recover informative image regions can harvest more discriminative representations, than to recover smooth image patches. Then we adopt a gradient based metric to estimate the importance of each image patch. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Moreover, we resort to the prior of brain structures and develop a Symmetric Position Encoding (SPE) method to better exploit the correlations between long-range but spatially symmetric regions to obtain effective features. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models on three brain MRI segmentation benchmarks.

show abstract

Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Cited by 377 publications

References 25 publications

Automatic Fetal Fat Quantification from MRI

Automatic Fetal Fat Quantification from MRI

Gait Recognition with Self-Supervised Learning of Gait Features Based on Vision Transformers

Attentive Symmetric Autoencoder for Brain MRI Segmentation

Contact Info

Product

Resources

About