NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

Abstract

Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities and apply modalitysensitive gating (MSG) at lower scales for more effective skip connections. Specifically, the multi-modal fusion is conducted in our proposed Nested Modality-aware Feature Aggregation (NMaFA) module, which enhances long-term dependencies within individual modalities via a triorientated spatial-attention transformer, and further complements key contextual information among modalities via a cross-modality attention transformer. Extensive experiments on BraTS2020 benchmark and a private meningiomas segmentation (MeniSeg) dataset show that the Nest-edFormer clearly outperforms the state-of-the-arts. The code is available at https://github.com/920232796/NestedFormer.

Understanding the brain with attention: A survey of transformers in brain sciences

“…In addition, some researchers abandoned U-shaped structures and built new Transformer architectures for more specific objectives. For example, Xing et al [137] proposed a Nested modality-aware TransFormer (NestedFormer), which focused on intra-and inter-modal relationships in multi-modal MRIs. NestedFormer adopted a multi-branch structure to extract features from different MRIs and then completed fusion in Transformer-based feature aggregation for more effective representation.…”

“…As discussed in Section 5.2, Transformers always load heavy parameters because of stacked attention modules and multi-head attention mechanics. Recent works have focused on some attempts to develop lightweight Transformers; for example, Chen [147] and Xing [137] et al shortened the attention modules in the case of sufficient semantics. In addition, Swin Transformers contribute to reducing parameters owing to their shift window-based attention.…”

“…In addition, some researchers abandoned U‐shaped structures and built new Transformer architectures for more specific objectives. For example, Xing et al [137] . proposed a Nested modality‐aware TransFormer (NestedFormer), which focused on intra‐ and inter‐modal relationships in multi‐modal MRIs.…”

Brain tumor segmentation and survival time prediction using graph momentum fully convolutional network with modified Elman spike neural network

“…For the analysis of BT, MRI is a critical diagnosis device 9 . To highlight various tissue properties of the BT, different complementary modalities like T1‐weighted (T1), T2‐weighted (T2), and so forth, based on 3D MRI are acquired 10–12 . The manual segmentation process is difficult and consumes more time 13,14 .…”

“…9 To highlight various tissue properties of the BT, different complementary modalities like T1-weighted (T1), T2-weighted (T2), and so forth, based on 3D MRI are acquired. [10][11][12] The manual segmentation process is difficult and consumes more time. 13,14 Thus that, automatic BTS enhances monitoring, treatment, and BT analysis, and it is beneficial.…”