nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer

Zhou, Hong-Yu; Guo, Jiansen; Zhang, Yinghao; Han, Xiaoguang; Yu, Lequan; Wang, Liansheng; Yu, Yizhou

doi:10.1109/tip.2023.3293771

Cited by 140 publications

(34 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared our full two-step segmentation pipeline with several standalone state-of-the-art segmentation algorithms, nnUNetv2, 54 Swin UNETR, 46 and nnFormer 68 . For extensive comparison, we obtained 15 additional test cases, and compared performance of all models on a total of 30 test cases.…”

Section: Resultsmentioning

confidence: 99%

Cascaded cross-attention transformers and convolutional neural networks for multi-organ segmentation in male pelvic computed tomography

Pemmaraju,

Kim,

Mekki

et al. 2024

J. Med. Imag.

View full text Add to dashboard Cite

Segmentation of the prostate and surrounding organs at risk from computed tomography is required for radiation therapy treatment planning. We propose an automatic two-step deep learning-based segmentation pipeline that consists of an initial multi-organ segmentation network for organ localization followed by organspecific fine segmentation.Approach: Initial segmentation of all target organs is performed using a hybrid convolutional-transformer model, axial cross-attention UNet. The output from this model allows for region of interest computation and is used to crop tightly around individual organs for organ-specific fine segmentation. Information from this network is also propagated to the fine segmentation stage through an image enhancement module, highlighting regions of interest in the original image that might be difficult to segment. Organ-specific fine segmentation is performed on these cropped and enhanced images to produce the final output segmentation. Results:We apply the proposed approach to segment the prostate, bladder, rectum, seminal vesicles, and femoral heads from male pelvic computed tomography (CT). When tested on a held-out test set of 30 images, our two-step pipeline outperformed other deep learning-based multi-organ segmentation algorithms, achieving average dice similarity coefficient (DSC) of 0.836 AE 0.071 (prostate), 0.947 AE 0.038 (bladder), 0.828 AE 0.057 (rectum), 0.724 AE 0.101 (seminal vesicles), and 0.933 AE 0.020 (femoral heads). Conclusions:Our results demonstrate that a two-step segmentation pipeline with initial multi-organ segmentation and additional fine segmentation can delineate male pelvic CT organs well. The utility of this additional layer of fine segmentation is most noticeable in challenging cases, as our two-step pipeline produces noticeably more accurate and less erroneous results compared to other state-of-the-art methods on such images.

show abstract

Section: Resultsmentioning

confidence: 99%

Cascaded cross-attention transformers and convolutional neural networks for multi-organ segmentation in male pelvic computed tomography

Pemmaraju,

Kim,

Mekki

et al. 2024

J. Med. Imag.

View full text Add to dashboard Cite

show abstract

“…To demonstrate the effectiveness of the proposed method, this paper was compared with TransUNet, 24 HiFormer, 25 nnUNet, 13 and nnFormer 26 multiple segmentation networks for the same segmentation task. The network segmentation results using the CT image data of gastric adenocarcinoma outlined by physicians are shown in Table 1.…”

Section: Resultsmentioning

confidence: 99%

Medical image segmentation of gastric adenocarcinoma based on dense connection of residuals

Hu,

Guo,

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

Background and objectiveAccurate segmentation of gastric cancer based on CT images of gastric adenocarcinoma is crucial for physicians to screen gastric diseases, clinical diagnosis, preoperative prediction, and postoperative evaluation plans. To address the issue of the inability of the segmentation algorithm to depict the correct boundaries due to unclear gastric contours in the lesion area and the visible irregular band‐like dense shadow extending to the perigastric region, a 3D medical image segmentation model 3D UNet based on residual dense jumping method is proposed.MethodsIn the method we proposed, Residual Dense Block, which is applied to the image super‐resolution module to remove CT artifacts, and Residual Block in ResNet are further fused. The quality of CT images is improved by Residual Dense Skip Block, which removes banded dense shadows, preserves image details and edge information, captures features, and improves the segmentation performance of gastric adenocarcinoma. The Instance Normalization layer position is modified to select the best result. Different loss functions are also combined in order to obtain the best gastric adenocarcinoma segmentation performance.ResultsWe tested the model on a hospital‐provided gastric adenocarcinoma dataset. The experimental results show that our model outperforms the existing methods in CT gastric adenocarcinoma segmentation, in which the method combining the hybrid loss function of Dice and CE obtains an average dice score of 82.3%, which is improved by 5.3% and 3.8% compared to TransUNet and Hiformer, respectively, and improves the cross‐merge rate to 70.8%, compared to nnFormer, nnUNet by 1% and 0.9%, respectively.ConclusionsThe residual jump connection structure indeed improves segmentation performance. The proposed method has the potential to be used as a screen for gastric diseases and to assist physicians in diagnosis.

show abstract

“…Li et al [28] proposed adaptive tokens to model the global context information and reduce computational complexity. Zhou et al [29] introduced a combination of self-attention mechanisms and interleaved convolutions and exploited local and global self-attention mechanisms to learn spatial features. Lee et al [30] set tokens of different sizes and input them into the Transformer through multiple paths.…”

Section: Vision Transformer-based Segmentation Methodsmentioning

confidence: 99%

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Jiao,

Yang,

Yan

et al. 2023

Electronics

View full text Add to dashboard Cite

Brain tumor image segmentation plays a significant auxiliary role in clinical diagnosis. Recently, deep learning has been introduced into multimodal segmentation tasks, which construct various Convolutional Neural Network (CNN) structures to achieve excellent performance. However, most CNN-based segmentation methods have poor capability for global feature extraction. Transformer is good at modeling long-distance dependencies, but it can cause local information loss and usually has a high computational complexity. In addition, it is difficult to fully exploit the brain tumor features of different modalities. To address these issues, in this paper, we propose a region–attention fusion (RAF) network that combines a dual-branch vision Transformer (DVT), called RFTNet. In RFTNet, the DVT is exploited to capture the delicate local information and global semantics separately by two branches. Meanwhile, a novel RAF is employed to effectively fuse the images of the different modalities. Finally, we design a new hybrid loss function, called region-mixed loss function (RML) to calculate the importance of each pixel and solve the problem of class imbalance. The experiments on BrasTS2018 and BraTS2020 datasets show that our method obtains a higher segmentation accuracy than other models. Furthermore, ablation experiments prove the effectiveness of each key component in RFTNet.

show abstract

nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer

Cited by 140 publications

References 23 publications

Cascaded cross-attention transformers and convolutional neural networks for multi-organ segmentation in male pelvic computed tomography

Cascaded cross-attention transformers and convolutional neural networks for multi-organ segmentation in male pelvic computed tomography

Medical image segmentation of gastric adenocarcinoma based on dense connection of residuals

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Contact Info

Product

Resources

About