A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction

Tan, Kaiwen; Huang, Weixian; Liu, Xiaofeng; Hu, Jinlong; Dong, Shen

doi:10.1016/j.artmed.2022.102260

Cited by 30 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding tumor subtype, the publication count increased, with 33 for histopathology and 26 for genomics. As anticipated, multimodality demonstrated the fewest publications, with three for diagnosis [ 23 , 24 ], two for grading [ 25 – 27 ], and two for subtyping [ 28 , 29 ]. Nevertheless, it is crucial to acknowledge that multimodal models present a significantly higher level of complexity compared to their unimodal counterparts.…”

Section: Resultsmentioning

confidence: 99%

A systematic analysis of deep learning in genomics and histopathology for precision oncology

Unger,

Kather

2024

BMC Med Genomics

View full text Add to dashboard Cite

Background Digitized histopathological tissue slides and genomics profiling data are available for many patients with solid tumors. In the last 5 years, Deep Learning (DL) has been broadly used to extract clinically actionable information and biological knowledge from pathology slides and genomic data in cancer. In addition, a number of recent studies have introduced multimodal DL models designed to simultaneously process both images from pathology slides and genomic data as inputs. By comparing patterns from one data modality with those in another, multimodal DL models are capable of achieving higher performance compared to their unimodal counterparts. However, the application of these methodologies across various tumor entities and clinical scenarios lacks consistency. Methods Here, we present a systematic survey of the academic literature from 2010 to November 2023, aiming to quantify the application of DL for pathology, genomics, and the combined use of both data types. After filtering 3048 publications, our search identified 534 relevant articles which then were evaluated by basic (diagnosis, grading, subtyping) and advanced (mutation, drug response and survival prediction) application types, publication year and addressed cancer tissue. Results Our analysis reveals a predominant application of DL in pathology compared to genomics. However, there is a notable surge in DL incorporation within both domains. Furthermore, while DL applied to pathology primarily targets the identification of histology-specific patterns in individual tissues, DL in genomics is more commonly used in a pan-cancer context. Multimodal DL, on the contrary, remains a niche topic, evidenced by a limited number of publications, primarily focusing on prognosis predictions. Conclusion In summary, our quantitative analysis indicates that DL not only has a well-established role in histopathology but is also being successfully integrated into both genomic and multimodal applications. In addition, there is considerable potential in multimodal DL for harnessing further advanced tasks, such as predicting drug response. Nevertheless, this review also underlines the need for further research to bridge the existing gaps in these fields.

show abstract

Section: Resultsmentioning

confidence: 99%

A systematic analysis of deep learning in genomics and histopathology for precision oncology

Unger,

Kather

2024

BMC Med Genomics

View full text Add to dashboard Cite

show abstract

“…Combining equations ( 3), (6), and ( 7), we can derive the final objective function. To simplify the format, we assume that:…”

Section: Optimization Of the Objective Equationmentioning

confidence: 99%

“…To improve multimodal fusion, researchers proposed kernel fusion methods that combined structural magnetic resonance imaging (sMRI), PET, and CSF data, 5 but these methods did not adequately consider the relationships between multimodal data. Subsequently, multitask learning methods were introduced to extract shared features from multimodal data, further enhancing fusion performance 6 . Some studies also employed techniques such as canonical correlation analysis (CCA), 7 but they were limited by sample constraints and issues with model generalization performance.…”

Section: Introductionmentioning

confidence: 99%

“…Subsequently, multitask learning methods were introduced to extract shared features from multimodal data, further enhancing fusion performance. 6 Some studies also employed techniques such as canonical correlation analysis (CCA), 7 but they were limited by sample constraints and issues with model generalization performance. Additionally, deep learning-based multimodal fusion methods have made advancements in the recognition of conditions like schizophrenia, mild cognitive impairment (MCI), 8 and AD, 9,10 however, these methods typically require large datasets and complex computations, making them challenging to apply in clinical support diagnosis for neurobehavioral disorders.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Research on multi‐center assisted diagnosis of ASD based on multimodal feature fusion

Zhang,

Sheng,

Wang

et al. 2024

Int J Imaging Syst Tech

View full text Add to dashboard Cite

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder, and structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) provide information about brain structure and function, aiding in objective ASD diagnosis. However, existing ASD classification methods face challenges such as sample scarcity, inter‐imaging center variations, insufficient single‐modality information, and inconsistent feature dimensions. This study introduced a method based on the Local Global Multimodal Domain Adaptation (LGMDA)‐Sparse Adaptive Prior Coupled Dictionary Learning (SACDL) framework. Initially, the LGMDA method was introduced to achieve multi‐source DA. By minimizing differences between different data domains while maximizing inter‐class differences within the same domain, it expands the sample size of multi‐modal data, addressing the issues of sample scarcity and heterogeneity in ASD data. Subsequently, the SACDL method was employed for multimodal fusion. It initialized dictionaries using the ATGP algorithm, combined sMRI and fMRI data for dictionary learning, adaptively adjusted sparsity parameters, and integrated ASD phenotype data for constrained optimization. It enables joint learning of shared and modality‐specific features, balancing differences in feature dimensions. Experimental results show that this model effectively utilizes multi‐center, multi‐modal information to achieve better auxiliary diagnosis than single‐modal small samples. This method has the potential to provide effective solutions for ASD multi‐source and multi‐modal classification problems, which are significant for ASD research and clinical diagnosis.

show abstract

“…8,9 In particular, without taking treatment into account, it is not possible to achieve a comparison between treatments to instruct the following treatment planning. 10 Although a possible solution is to have different direct models for each treatment, there is only a limited amount of training data that can be used for each model.…”

Section: Introductionmentioning

confidence: 99%

Treatment-wise glioblastoma survival Inference with multi-parametric preoperative MRI

Liu,

Shusharina,

Shih

et al. 2024

Medical Imaging 2024: Computer-Aided Diagnosis

View full text Add to dashboard Cite

In this work, we aim to predict the survival time (ST) of glioblastoma (GBM) patients undergoing different treatments based on preoperative magnetic resonance (MR) scans. The personalized and precise treatment planning can be achieved by comparing the ST of different treatments. It is well established that both the current status of the patient (as represented by the MR scans) and the choice of treatment are the cause of ST. While previous related MR-based glioblastoma ST studies have focused only on the direct mapping of MR scans to ST, they have not included the underlying causal relationship between treatments and ST. To address this limitation, we propose a treatment-conditioned regression model for glioblastoma ST that incorporates treatment information in addition to MR scans. Our approach allows us to effectively utilize the data from all of the treatments in a unified manner, rather than having to train separate models for each of the treatments. Furthermore, treatment can be effectively injected into each convolutional layer through the adaptive instance normalization we employ. We evaluate our framework on the BraTS20 ST prediction task. Three treatment options are considered: Gross Total Resection (GTR), Subtotal Resection (STR), and no resection. The evaluation results demonstrate the effectiveness of injecting the treatment for estimating GBM survival.

show abstract

A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction

Cited by 30 publications

References 28 publications

A systematic analysis of deep learning in genomics and histopathology for precision oncology

A systematic analysis of deep learning in genomics and histopathology for precision oncology

Research on multi‐center assisted diagnosis of ASD based on multimodal feature fusion

Treatment-wise glioblastoma survival Inference with multi-parametric preoperative MRI

Contact Info

Product

Resources

About