Artificial intelligence for multimodal data integration in oncology

Lipková, Jana; Chen, Richard J.; Chen, Bowen; Lu, Ming; Barbieri, Matteo; Shao, Daniel; Vaidya, Anurag; Chen, Chengkuan; Zhuang, Luoting; Williamson, Drew F. K.; Shaban, Muhammad; Chen, Tiffany; Mahmood, Faisal

doi:10.1016/j.ccell.2022.09.012

Cited by 261 publications

(126 citation statements)

References 101 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…Notably, for some cancers there were significant performance differences between using sequence context or gene information, suggesting combining these two features would give the best performance. If instance features are suspected to interact with each other they should clearly be fused at the instance level, but if no interaction is expected there is a range of possibilities for combining the features 24,25 . While careful consideration should be made on how to best fuse input features in the context of MIL, that was not the focus of this study and the best approach may be data dependent.…”

Section: Resultsmentioning

confidence: 99%

Aggregation Tool for Genomic Concepts (ATGC): A deep learning framework for somatic mutations and other sparse genomic measures

Anaya

Sidhom

Cummings

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning has the ability to extract meaningful features from data given enough training examples. Large scale genomic data are well suited for this class of machine learning algorithms; however, for many of these data the labels are at the level of the sample instead of at the level of the individual genomic measures. To leverage the power of deep learning for these types of data we turn to a multiple instance learning framework, and present an easily extensible tool built with TensorFlow and Keras. We show how this tool can be applied to somatic variants (featurizing genomic position and sequence context), and accurately classify samples according to whether they contain a specific variant (hotspot or tumor suppressor) or whether they contain a type of variant (microsatellite instability). We then apply our model to the calibration of tumor mutational burden (TMB), an increasingly important metric in the field of immunotherapy, across a variety of commonly used gene panels. Regardless of the panel, we observed improvements in regression to the gold standard whole exome derived value for this metric, with additional performance benefits as more data were provided to the model (such as noncoding variants from panel assays). Our results suggest this framework could lead to improvements in a range of tasks where the sample level metric is determined by the aggregation of a set of genomic measures, such as somatic mutations that we focused on in this study.

show abstract

Section: Resultsmentioning

confidence: 99%

Aggregation Tool for Genomic Concepts (ATGC): A deep learning framework for somatic mutations and other sparse genomic measures

Anaya

Sidhom

Cummings

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The use of multiple imaging modalities to evaluate bone tumours is known to improve the accuracy of diagnosis [ 3 ]. Machine learning models built using different modalities (multimodal) have also been shown to improve diagnostic performance [ 193 ]. In breast radiology, Antropova et al [ 194 ] came out with a CNN method involving fusion-based classification using dynamic contrast enhanced-MRI, full-field digital mammography, and ultrasound.…”

Section: Discussionmentioning

confidence: 99%

Application of Machine Learning for Differentiating Bone Malignancy on Imaging: A Systematic Review

Ong

Zhu

Tan

et al. 2023

Cancers

View full text Add to dashboard Cite

An accurate diagnosis of bone tumours on imaging is crucial for appropriate and successful treatment. The advent of Artificial intelligence (AI) and machine learning methods to characterize and assess bone tumours on various imaging modalities may assist in the diagnostic workflow. The purpose of this review article is to summarise the most recent evidence for AI techniques using imaging for differentiating benign from malignant lesions, the characterization of various malignant bone lesions, and their potential clinical application. A systematic search through electronic databases (PubMed, MEDLINE, Web of Science, and clinicaltrials.gov) was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. A total of 34 articles were retrieved from the databases and the key findings were compiled and summarised. A total of 34 articles reported the use of AI techniques to distinguish between benign vs. malignant bone lesions, of which 12 (35.3%) focused on radiographs, 12 (35.3%) on MRI, 5 (14.7%) on CT and 5 (14.7%) on PET/CT. The overall reported accuracy, sensitivity, and specificity of AI in distinguishing between benign vs. malignant bone lesions ranges from 0.44–0.99, 0.63–1.00, and 0.73–0.96, respectively, with AUCs of 0.73–0.96. In conclusion, the use of AI to discriminate bone lesions on imaging has achieved a relatively good performance in various imaging modalities, with high sensitivity, specificity, and accuracy for distinguishing between benign vs. malignant lesions in several cohort studies. However, further research is necessary to test the clinical performance of these algorithms before they can be facilitated and integrated into routine clinical practice.

show abstract

“…Deep learning models are particularly effective and leverage artificial neural networks (ANNs) to make predictions based on input data [3]. As data availability has proliferated through large-scale consortiums such as The Cancer Genome Atlas (TCGA), ML techniques have been applied to integrate a wide range of input data modalities for prognostication [4], such as whole-slideimaging (WSI), gene expression quantification, clinical attributes like age and gender, and other biological and molecular measures [5][6][7]. These methods aim to quantify prognosis as a scalar hazard ratio and are supervised by Cox loss, a common method in survival analysis [8].…”

Section: Background Cancer Prognostication and Machine Learningmentioning

confidence: 99%

Assessment of Emerging Pretraining Strategies in Interpretable Multimodal Deep Learning for Cancer Prognostication

Azher¹,

Suvarna²,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning models have demonstrated the remarkable ability to infer cancer patient prognosis from molecular and anatomic pathology information. Studies in recent years have demonstrated that leveraging information from complementary multimodal data can improve prognostication, further illustrating the potential utility of such methods. Model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology. However, while prior works have presented novel multimodal neural network architectures as means to improve prognostication performance, these approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to "pretrain" models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. Here, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare the performances of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure in this context. Our models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models. Our results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression.

show abstract

Artificial intelligence for multimodal data integration in oncology

Cited by 261 publications

References 101 publications

Aggregation Tool for Genomic Concepts (ATGC): A deep learning framework for somatic mutations and other sparse genomic measures

Aggregation Tool for Genomic Concepts (ATGC): A deep learning framework for somatic mutations and other sparse genomic measures

Application of Machine Learning for Differentiating Bone Malignancy on Imaging: A Systematic Review

Assessment of Emerging Pretraining Strategies in Interpretable Multimodal Deep Learning for Cancer Prognostication

Contact Info

Product

Resources

About