More than Encoder: Introducing Transformer Decoder to Upsample

Li, Yijiang; Cai, Wentian; Hu, Xiping

doi:10.48550/arxiv.2106.10637

Cited by 6 publications

(6 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xu et al [38] extensively studied the trade-off between transformers and convnets and proposed a more efficient encoder named LeViT-UNet. Li et al [17] presented a new up-sampling approach and incorporated it into the decoder of UNet to model long-term dependencies and global information for better reconstruction results. TransClaw U-Net [4] utilized transformers in UNet with more convolutional feature pyramids.…”

Section: Related Workmentioning

confidence: 99%

nnFormer: Interleaved Transformer for Volumetric Segmentation

Zhou¹,

Guo²,

Zhang³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

Transformers, the default model of choices in natural language processing, have drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks (convnets) to overcome its inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations without investigating how to optimally combine self-attention (i.e., the core of transformers) with convolution. To address this issue, in this paper, we introduce nnFormer (i.e., not-another transFormer), a powerful segmentation model with an interleaved architecture based on empirical combination of self-attention and convolution. In practice, nnFormer learns volumetric representations from 3D local volumes. Compared to the naive voxel-level self-attention implementation, such volume-based operations help to reduce the computational complexity by approximate 98% and 99.5% on Synapse and ACDC datasets, respectively. In comparison to prior-art network configurations, nnFormer achieves tremendous improvements over previous transformer-based methods on two commonly used datasets Synapse and ACDC. For instance, nnFormer outperforms Swin-UNet by over 7 percents on Synapse. Even when compared to nnUNet, currently the best performing fully-convolutional medical segmentation network, nnFormer still provides slightly better performance on Synapse and ACDC. Codes and models are available at https://github.com/282857341/nnFormer.

show abstract

Section: Related Workmentioning

confidence: 99%

nnFormer: Interleaved Transformer for Volumetric Segmentation

Zhou¹,

Guo²,

Zhang³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

show abstract

“…The Medical imaging field started adopting transformers right away, Implementing them in different applications for diagnosis and prognosis, He et al (2022) 9 present a comprehensive review of many applications using transformer-based architectures in the medical imaging field. For segmentation, transformer-based architectures were used for example, in cardiac segmentation, 10,11,12,13,14 achieved state-of-the-art results on different related datasets, while in multi-organ segmentation 10,11,13,14,15,16,17 also achieved state-of-the-art results.…”

Section: Vision Transformersmentioning

confidence: 99%

NesT UNet: pure transformer segmentation network with an application for automatic cardiac myocardial infarction evaluation

Abdelhamed¹,

Mériaudeau²

2023

Medical Imaging 2023: Computer-Aided Diagnosis

View full text Add to dashboard Cite

Myocardial Infarction (MI), commonly known as heart attack, is the irreversible death of the Myocardium's tissue due to oxygen deprivation for an extended period of time. LGE-MRI scans are considered the defacto in the diagnosis and prognosis of MI. Still, they require manually segmenting the Myocardium and the infarcted tissue, which is a complex and time-consuming task. Hence, an automatic segmentation method of the Myocardium tissue is highly desirable. CNNs (Convolutional Neural Networks) are used extensively in cardiac tissue segmentation in general and for solving this problem particularly. EMIDEC (automatic Evaluation of Myocardial Infarction from Delayed Enhancement Cardiac MRI) challenge in MICCAI 2020 provides a good overview of the different CNN-based architectures used for MI segmentation. Still, they required complex pipelines to achieve state-of-the-art results and the research followed the challenge tends to go in the same direction of building more complex pipelines revolving around the famous CNN-UNet either 2D or 3D. In this paper, we present a different direction by presenting a simple 2D novel architecture based on Self-Attention Transformers offering a possible alternative to the CNN-UNet as a building block for bigger systems. We introduce NesT-UNet a novel segmentation architecture based on the NesT architecture as an encoder and we also introduce a novel decoder inspired by the same architecture. NesT achieved state-of-the-art results on ImageNet and CIFAR classification tasks with minimal training compared to other transformer networks. 2D NesT-UNet produced results comparable to the state-of-the-art on the EMIDEC dataset using a simple training process and an extra self-supervised pre-training step to improve the network's performance. we also present a novel loss function for false positives reduction.

show abstract

“…Using well-known data sets related to different use cases and working on top of [52], Roy et al [114] verified that the addition of a spatial-channel squeeze and excitation block works as an attention mechanism in FCNs and improves the quality of the segmentation maps. Interestingly, in most recent papers, the authors approach the problem of medical image segmentation using encoder-decoder structures that generally comprise a CNN and a Transformer to extract volumetric spatial feature maps, perform global feature modeling and predict refined segmentation maps [115][116][117][118], although there are already some works that replace the CNN-based modules at the encoder or decoder levels and integrate other attention mechanisms to extract features and model longrange dependencies [119]. More recent methodologies on medical image segmentation are taking advantage of a hybrid use of the Vision Transformer and the U-Net with improved results regarding the quality of segmentation maps [120,121].…”

Section: B Medical Image Segmentationmentioning

confidence: 99%

A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?

et al. 2022

View full text Add to dashboard Cite

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning methods (including Transformers) for several medical applications based on the types of tasks that may integrate several works pipelines of the medical domain. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.

show abstract

More than Encoder: Introducing Transformer Decoder to Upsample

Cited by 6 publications

References 56 publications

nnFormer: Interleaved Transformer for Volumetric Segmentation

nnFormer: Interleaved Transformer for Volumetric Segmentation

NesT UNet: pure transformer segmentation network with an application for automatic cardiac myocardial infarction evaluation

A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?

Contact Info

Product

Resources

About