More than Encoder: Introducing Transformer Decoder to Upsample

Li, Yijiang; Cai, Wentian; Li, Chengming; Hu, Xiping

doi:10.1109/bibm55620.2022.9995378

Cited by 23 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[34] Inspired by Swin Transformer, [48] Hatamizadeh et al [55] designed the Swin UNETR model and achieved Dice scores of 85.30%, 92.70%, and 87.60% for enhancing tumors (ET), whole tumors (WT), and tumor core (TC) respectively on the Multi-modal Brain Tumor Segmentation Challenge (BraTS2021) dataset. [57] Recognizing the impact of kernel design on performance, Li et al [56] introduced a Transformer decoder with window attention decoding and achieved Dice values of 63.23%, 80.29%, and 80.73% for non-enhanced tumors, ET, and cerebral edema respectively on the MSD brain dataset. [57] Zhang et al [62] incorporated filtering feature integration mechanisms and improved star-shaped window (S 2 WIN) Transformers in the decoder to achieve better boundary delineation.…”

Section: Brain Tumor Detectionmentioning

confidence: 99%

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

Cong,

Wang,

Zhou

et al. 2024

Brain-X

View full text Add to dashboard Cite

This comprehensive review aims to clarify the growing impact of Transformer‐based models in the fields of neuroscience, neurology, and psychiatry. Originally developed as a solution for analyzing sequential data, the Transformer architecture has evolved to effectively capture complex spatiotemporal relationships and long‐range dependencies that are common in biomedical data. Its adaptability and effectiveness in deciphering intricate patterns within medical studies have established it as a key tool in advancing our understanding of neural functions and disorders, representing a significant departure from traditional computational methods. The review begins by introducing the structure and principles of Transformer architectures. It then explores their applicability, ranging from disease diagnosis and prognosis to the evaluation of cognitive processes and neural decoding. The specific design modifications tailored for these applications and their subsequent impact on performance are also discussed. We conclude by providing a comprehensive assessment of recent advancements, prevailing challenges, and future directions, highlighting the shift in neuroscientific research and clinical practice towards an artificial intelligence‐centric paradigm, particularly given the prominence of Transformer architecture in the most successful large pre‐trained models. This review serves as an informative reference for researchers, clinicians, and professionals who are interested in understanding and harnessing the transformative potential of Transformer‐based models in neuroscience, neurology, and psychiatry.

show abstract

Section: Brain Tumor Detectionmentioning

confidence: 99%

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

Cong,

Wang,

Zhou

et al. 2024

Brain-X

View full text Add to dashboard Cite

show abstract

“…Subsequently, Hatamizadeh et al [23] proposed the Swin UNETR, which employs hierarchical Swin Transformer blocks as the encoder and ranked first in the BraTS 2021 Challenge validation phase. Li et al [24] proposed Window Attention Up-sample (WAU) to increase the sampling of features in the decoder path by Transformer attention decoders. Pham et al [25] used a Transformer with a variational autoencoder (VAE) branch to reconstruct input images concurrently with segmentation.…”

Section: Transformer-based Brain Tumor Segmentation Modelsmentioning

confidence: 99%

Clinical Knowledge-Based Hybrid Swin Transformer for Brain Tumor Segmentation

Lei,

Yu,

et al. 2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Accurate tumor segmentation from brain tissues in Magnetic Resonance Imaging (MRI) imaging is crucial in the pre-surgical planning of brain tumor malignancy. MRI images' heterogeneous intensity and fuzzy boundaries make brain tumor segmentation challenging. Furthermore, recent studies have yet to fully employ MRI sequences' considerable and supplementary information, which offers critical a priori knowledge. This paper proposes a clinical knowledge-based hybrid Swin Transformer multimodal brain tumor segmentation algorithm based on how experts identify malignancies from MRI images. During the encoder phase, a dual backbone network with a Swin Transformer backbone to capture long dependencies from 3D MR images and a Convolutional Neural Network (CNN)-based backbone to represent local features have been constructed. Instead of directly connecting all the MRI sequences, the proposed method re-organizes them and splits them into two groups based on MRI principles and characteristics: T1 and T1ce, T2 and Flair. These aggregated images are received by the dual-stem Swin Transformer-based encoder branch, and the multimodal sequence-interacted cross-attention module (MScAM) captures the interactive information between two sets of linked modalities in each stage. In the CNN-based encoder branch, a triple down-sampling module (TDsM) has been proposed to balance the performance while downsampling. In the final stage of the encoder, the feature maps acquired from two branches are concatenated as input to the decoder, which is constrained by MScAM outputs. The proposed method has been evaluated on datasets from the MICCAI BraTS2021 Challenge. The results of the experiments demonstrate that the method algorithm can precisely segment brain tumors, especially the portions within tumors.

show abstract

“…It is often used for i age super-resolution [20], segmentation [21], and generation [22] tasks via the reconstr tion of high-resolution feature maps during the decoding stage in the neural network [2 Upsampling also plays an important role in neural networks. It is often used for image super-resolution [20], segmentation [21], and generation [22] tasks via the reconstruction of high-resolution feature maps during the decoding stage in the neural network [23]. The main upsampling methods include interpolation-based upsampling such as the Nearest Neighbor, Bilinear, and Bicubic Interpolation methods [24] and the Transposed Convolution [25] and Sub-Pixel Convolutional [26] methods.…”

Section: Introductionmentioning

confidence: 99%

SliceSamp: A Promising Downsampling Alternative for Retaining Information in a Neural Network

He,

Wang

2023

Applied Sciences

View full text Add to dashboard Cite

Downsampling, which aims to improve computational efficiency by reducing the spatial resolution of feature maps, is a critical operation in neural networks. Many downsampling methods have been proposed to address the challenge of retaining feature map information. However, some detailed information is still lost, even though these methods can extract features with stronger semantics. In this paper, we propose a novel downsampling method which combines feature slicing and depthwise separable convolution for information-retaining downsampling. It slices the input feature map into multiple non-overlapping sub-feature maps by using indexes with a stride of two in the spatial dimension and applies depthwise separable convolution on each slice to extract feature information. To demonstrate the effectiveness of SliceSamp, we compare it with classical downsampling methods on image classification, object detection, and semantic segmentation tasks using several benchmark datasets, including ImageNet-1K, COCO, VOC, and ADE20K. Extensive experiments demonstrate that SliceSamp outperforms classical downsampling methods with consistent improvements in various computer vision tasks. The proposed SliceSamp shows advanced model performance with lower computational costs and memory requirements. By replacing the downsampling layers in different network architectures (including ResNet (Residual Network), YOLOv5, and Swin Transformer), SliceSamp brings different degrees of performance gains (+0.54~3.64%) compared to these baseline models. Additionally, SliceUpsamp enables high-resolution feature reconstruction and alignment during upsampling. SliceSamp and SliceUpsamp can be plug-and-play-integrated into existing neural network architectures. As a promising downsampling alternative to traditional methods, SliceSamp can also provide a reference for designing lightweight and high-performance model architectures in the future.

show abstract

More than Encoder: Introducing Transformer Decoder to Upsample

Cited by 23 publications

References 18 publications

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

Clinical Knowledge-Based Hybrid Swin Transformer for Brain Tumor Segmentation

SliceSamp: A Promising Downsampling Alternative for Retaining Information in a Neural Network

Contact Info

Product

Resources

About