A Fully Automated Multimodal MRI-Based Multi-Task Learning for Glioma Segmentation and IDH Genotyping

Cheng, Jianhong; Liu, Jin; Kuang, Hulin; Wang, Jianxin

doi:10.1109/tmi.2022.3142321

Cited by 94 publications

(46 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Ken’s study [ 15 ], T2 image-based external testing achieved AUC = 0.73 and ACC = 67.3%, which was inferior to our results. Besides the CNN-based studies, only one study introduced the transformer to the IDH genotyping [ 32 ] without external testing, achieving an internal TCIA test of AUC = 91.04% and ACC = 90%, which was lower than our internal test results. Our Swin Transformer network with bounding box inputs showed great potential in IDH mutation prediction.…”

Section: Discussionmentioning

confidence: 76%

“…However, new DL architectures, such as Transformer, have been seldom introduced to perform the IDH prediction. Transformer, a novel neural architecture whose empirical performance significantly outperforms the conventional CNNs, can effectively capture long-range contextual relations between image pixels and approach to be a state-of-the-art network for medical image representation [ 29 , 30 , 31 , 32 ]. Until now, only one study has applied this framework to IDH mutation status prediction using the TCIA dataset [ 32 ], and more research needs to be performed to demonstrate its generalization and compassion to CNNs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Shen

et al. 2022

JCM

View full text Add to dashboard Cite

Background: Deep learning (DL) could predict isocitrate dehydrogenase (IDH) mutation status from MRIs. Yet, previous work focused on CNNs with refined tumor segmentation. To bridge the gap, this study aimed to evaluate the feasibility of developing a Transformer-based network to predict the IDH mutation status free of refined tumor segmentation. Methods: A total of 493 glioma patients were recruited from two independent institutions for model development (TCIA; N = 259) and external test (AHXZ; N = 234). IDH mutation status was predicted directly from T2 images with a Swin Transformer and conventional ResNet. Furthermore, to investigate the necessity of refined tumor segmentation, seven strategies for the model input image were explored: (i) whole tumor slice; (ii–iii) tumor mask and/or not edema; (iv–vii) tumor bounding box of 0.8, 1.0, 1.2, 1.5 times. Performance comparison was made among the networks of different architectures along with different image input strategies, using area under the curve (AUC) and accuracy (ACC). Finally, to further boost the performance, a hybrid model was built by incorporating the images with clinical features. Results: With the seven proposed input strategies, seven Swin Transformer models and seven ResNet models were built, respectively. Based on the seven Swin Transformer models, an averaged AUC of 0.965 (internal test) and 0.842 (external test) were achieved, outperforming 0.922 and 0.805 resulting from the seven ResNet models, respectively. When a bounding box of 1.0 times was used, Swin Transformer (AUC = 0.868, ACC = 80.7%), achieved the best results against the one that used tumor segmentation (Tumor + Edema, AUC = 0.862, ACC = 78.5%). The hybrid model that integrated age and location features into images yielded improved performance (AUC = 0.878, Accuracy = 82.0%) over the model that used images only. Conclusions: Swin Transformer outperforms the CNN-based ResNet in IDH prediction. Using bounding box input images benefits the DL networks in IDH prediction and makes the IDH prediction free of refined glioma segmentation feasible.

show abstract

Section: Discussionmentioning

confidence: 76%

Section: Introductionmentioning

confidence: 99%

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Shen

et al. 2022

JCM

View full text Add to dashboard Cite

show abstract

“…To overcome the heterogeneity of glioma, several AI algorithm application tools have been used. In particular, combining imaging techniques (i.e., CT, MRI, PET-CT) with metabolic markers and proteome data has been shown to yield useful information for clinical applications [ 69 ]. Current models predict overall survival, progression-free survival, and molecular subtypes of high-grade glioma as well as genetic alterations.…”

Section: Clinical and Management Features Of Significance In Gliomamentioning

confidence: 99%

“…Previous research has used transcriptomic data to develop prognostic models for GBM patients [ 75 ]. Machine learning prognostic models using a combination of imaging patterns, clinical features, molecular markers, and radiomic data have shown promising results [ 69 , 95 , 96 ]. The rational implementation of AI algorithms—especially in areas such as imaging where data are already generated as part of the standard of care, as is the case with MRI of the brain for glioma patients—are crucial, since this forms the “floor” or base from which diagnosis and management originate in glioma and are already built into the cost of care ( Figure 3 ).…”

Section: Optimizing Cost-benefit In Glioma To Advance Patient Outcomesmentioning

confidence: 99%

Cost Matrix of Molecular Pathology in Glioma—Towards AI-Driven Rational Molecular Testing and Precision Care for the Future

et al. 2022

View full text Add to dashboard Cite

Gliomas are the most common and aggressive primary brain tumors. Gliomas carry a poor prognosis because of the tumor’s resistance to radiation and chemotherapy leading to nearly universal recurrence. Recent advances in large-scale genomic research have allowed for the development of more targeted therapies to treat glioma. While precision medicine can target specific molecular features in glioma, targeted therapies are often not feasible due to the lack of actionable markers and the high cost of molecular testing. This review summarizes the clinically relevant molecular features in glioma and the current cost of care for glioma patients, focusing on the molecular markers and meaningful clinical features that are linked to clinical outcomes and have a realistic possibility of being measured, which is a promising direction for precision medicine using artificial intelligence approaches.

show abstract

“…In the literature, some recent studies [6,16,27] have been presented to address the aforementioned issue via joint representation learning from multi-modalities. They directly align the feature representation of corresponding pixels from different modalities during the training phase.…”

Section: Introductionmentioning

confidence: 99%

Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency

Yang¹,

Zhu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Integrating multi-modal data to improve medical image analysis has received great attention recently. However, due to the modal discrepancy, how to use a single model to process the data from multiple modalities is still an open issue. In this paper, we propose a novel scheme to achieve better pixel-level segmentation for unpaired multi-modal medical images. Different from previous methods which adopted both modality-specific and modality-shared modules to accommodate the appearance variance of different modalities while extracting the common semantic information, our method is based on a single Transformer with a carefully designed External Attention Module (EAM) to learn the structured semantic consistency (i.e. semantic class representations and their correlations) between modalities in the training phase. In practice, the above-mentioned structured semantic consistency across modalities can be progressively achieved by implementing the consistency regularization at the modality-level and image-level respectively. The proposed EAMs are adopted to learn the semantic consistency for different scale representations and can be discarded once the model is optimized. Therefore, during the testing phase, we only need to maintain one Transformer for all modal predictions, which nicely balances the model's ease of use and simplicity. To demonstrate the effectiveness of the proposed method, we conduct the experiments on two medical image segmentation scenarios: (1) cardiac structure segmentation, and (2) abdominal multi-organ segmentation. Extensive results show that the proposed method outperforms the state-of-the-art methods by a wide margin, and even achieves competitive performance with extremely limited training samples (e.g., 1 or 3 annotated CT or MRI images ) for one specific modality.

show abstract

A Fully Automated Multimodal MRI-Based Multi-Task Learning for Glioma Segmentation and IDH Genotyping

Cited by 94 publications

References 41 publications

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Cost Matrix of Molecular Pathology in Glioma—Towards AI-Driven Rational Molecular Testing and Precision Care for the Future

Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency

Contact Info

Product

Resources

About