Multimodal data fusion for cancer biomarker discovery with deep learning

Steyaert, Sandra; Pizurica, Marija; Nagaraj, Divya; Khandelwal, Priya; Hernandez‐Boussard, Tina; Gentles, Andrew J.; Gevaert, Olivier

doi:10.1038/s42256-023-00633-5

Cited by 91 publications

(26 citation statements)

References 154 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the present study, we revealed that the multi-modality analysis reached higher performance than CNNs in predicting MGMT promoter methylation although vViT could not achieve state-of-the-art performance. The development of effective multimodal fusion approaches is becoming increasingly important to capture features of complex diseases [ 39 ]. Predicting MGMT promoter methylation among adult patients with diffuse glioma is not an exception.…”

Section: Discussionmentioning

confidence: 99%

Identifying key factors for predicting O6-Methylguanine-DNA methyltransferase status in adult patients with diffuse glioma: a multimodal analysis of demographics, radiomics, and MRI by variable Vision Transformer

Usuzaki,

Takahashi,

Inamori

et al. 2024

Neuroradiology

View full text Add to dashboard Cite

Purpose This study aimed to perform multimodal analysis by vision transformer (vViT) in predicting O6-methylguanine-DNA methyl transferase (MGMT) promoter status among adult patients with diffuse glioma using demographics (sex and age), radiomic features, and MRI. Methods The training and test datasets contained 122 patients with 1,570 images and 30 patients with 484 images, respectively. The radiomic features were extracted from enhancing tumors (ET), necrotic tumor cores (NCR), and the peritumoral edematous/infiltrated tissues (ED) using contrast-enhanced T1-weighted images (CE-T1WI) and T2-weighted images (T2WI). The vViT had 9 sectors; 1 demographic sector, 6 radiomic sectors (CE-T1WI ET, CE-T1WI NCR, CE-T1WI ED, T2WI ET, T2WI NCR, and T2WI ED), 2 image sectors (CE-T1WI, and T2WI). Accuracy and area under the curve of receiver-operating characteristics (AUC-ROC) were calculated for the test dataset. The performance of vViT was compared with AlexNet, GoogleNet, VGG16, and ResNet by McNemar and Delong test. Permutation importance (PI) analysis with the Mann–Whitney U test was performed. Results The accuracy was 0.833 (95% confidence interval [95%CI]: 0.714–0.877) and the area under the curve of receiver-operating characteristics was 0.840 (0.650–0.995) in the patient-based analysis. The vViT had higher accuracy than VGG16 and ResNet, and had higher AUC-ROC than GoogleNet (p<0.05). The ED radiomic features extracted from the T2-weighted image demonstrated the highest importance (PI=0.239, 95%CI: 0.237–0.240) among all other sectors (p<0.0001). Conclusion The vViT is a competent deep learning model in predicting MGMT status. The ED radiomic features of the T2-weighted image demonstrated the most dominant contribution.

show abstract

Section: Discussionmentioning

confidence: 99%

Identifying key factors for predicting O6-Methylguanine-DNA methyltransferase status in adult patients with diffuse glioma: a multimodal analysis of demographics, radiomics, and MRI by variable Vision Transformer

Usuzaki,

Takahashi,

Inamori

et al. 2024

Neuroradiology

View full text Add to dashboard Cite

show abstract

“…Multimodal fusion is a technique used to complement multiple heterogeneous information extracted from different modalities of drugs and targets. There are three commonly used multimodal fusion strategies: early fusion, late fusion, and hybrid fusion …”

Section: Methodsmentioning

confidence: 99%

“…There are three commonly used multimodal fusion strategies: early fusion, late fusion, and hybrid fusion. 31 Early fusion involves extracting heterogeneous information from the original different modalities and then fusing this information. There are three common methods for early fusion: Addition, Concatenation, and Multiplication.…”

Section: Modal Information Representationmentioning

confidence: 99%

MMDTA: A Multimodal Deep Model for Drug-Target Affinity with a Hybrid Fusion Strategy

Zhong,

Wen,

Meng

et al. 2023

J. Chem. Inf. Model.

View full text Add to dashboard Cite

The prediction of the drug-target affinity (DTA) plays an important role in evaluating molecular druggability. Although deep learning-based models for DTA prediction have been extensively attempted, there are rare reports on multimodal models that leverage various fusion strategies to exploit heterogeneous information from multiple different modalities of drugs and targets. In this study, we proposed a multimodal deep model named MMDTA, which integrated the heterogeneous information from various modalities of drugs and targets using a hybrid fusion strategy to enhance DTA prediction. To achieve this, MMDTA first employed convolutional neural networks (CNNs) and graph convolutional networks (GCNs) to extract diverse heterogeneous information from the sequences and structures of drugs and targets. It then utilized a hybrid fusion strategy to combine and complement the extracted heterogeneous information, resulting in the fused modal information for predicting drugtarget affinity through the fully connected (FC) layers. Experimental results demonstrated that MMDTA outperformed the competitive state-of-the-art deep learning models on the widely used benchmark data sets, particularly with a significantly improved key evaluation metric, Root Mean Square Error (RMSE). Furthermore, MMDTA exhibited excellent generalization and practical application performance on multiple different data sets. These findings highlighted MMDTA's accuracy and reliability in predicting the drug-target binding affinity. For researchers interested in the source data and code, they are accessible at http://github.com/ dldxzx/MMDTA.

show abstract

“…Comprehensive information on developing machine intelligence in oncology could be found in contemporary reviews of the state of the art. [66][67][68][69][70][71] Successful models represent a Software-as-Medical-Device (SaMD) endpoint 72 that could then be refined with real-time clinical data for continuous deployment and integration. Examples of such models built using miRNA biomarkers have been recently reported.…”

Section: Construction Of ML Modelmentioning

confidence: 99%

Microfluidics for Profiling miRNA Biomarker Panels in AI-Assisted Cancer Diagnosis and Prognosis

Muthamilselvan

Baabu

Palaniappan

2023

Technol Cancer Res Treat

View full text Add to dashboard Cite

Early detection of cancers and their precise subtyping are essential to patient stratification and effective cancer management. Data-driven identification of expression biomarkers coupled with microfluidics-based detection shows promise to revolutionize cancer diagnosis and prognosis. MicroRNAs play key roles in cancers and afford detection in tissue and liquid biopsies. In this review, we focus on the microfluidics-based detection of miRNA biomarkers in AI-based models for early-stage cancer subtyping and prognosis. We describe various subclasses of miRNA biomarkers that could be useful in machine-based predictive modeling of cancer staging and progression. Strategies for optimizing the feature space of miRNA biomarkers are necessary to obtain a robust signature panel. This is followed by a discussion of the issues in model construction and validation towards producing Software-as-Medical-Devices (SaMDs). Microfluidic devices could facilitate the multiplexed detection of miRNA biomarker panels, and an overview of the different strategies for designing such microfluidic systems is presented here, with an outline of the detection principles used and the corresponding performance measures. Microfluidics-based profiling of miRNAs coupled with SaMD represent high-performance point-of-care solutions that would aid clinical decision-making and pave the way for accessible precision personalized medicine.

show abstract

Multimodal data fusion for cancer biomarker discovery with deep learning

Cited by 91 publications

References 154 publications

Identifying key factors for predicting O6-Methylguanine-DNA methyltransferase status in adult patients with diffuse glioma: a multimodal analysis of demographics, radiomics, and MRI by variable Vision Transformer

Identifying key factors for predicting O6-Methylguanine-DNA methyltransferase status in adult patients with diffuse glioma: a multimodal analysis of demographics, radiomics, and MRI by variable Vision Transformer

MMDTA: A Multimodal Deep Model for Drug-Target Affinity with a Hybrid Fusion Strategy

Microfluidics for Profiling miRNA Biomarker Panels in AI-Assisted Cancer Diagnosis and Prognosis

Contact Info

Product

Resources

About