Multimodal Fusion Strategies for Outcome Prediction in Stroke

Abstract

Data driven methods are increasingly being adopted in the medical domain for clinical predictive modeling. Prediction of stroke outcome using machine learning could provide a decision support system for physicians to assist them in patient-oriented diagnosis and treatment. While patient-specific clinical parameters play an important role in outcome prediction, a multimodal fusion approach that integrates neuroimaging with clinical data has the potential to improve accuracy. This paper addresses two research questions: (a) does multimodal fusion aid in the prediction of stroke outcome, and (b) what fusion strategy is more suitable for the task at hand. The baselines for our experimental work are two unimodal neural architectures: a 3D Convolutional Neural Network for processing neuroimaging data, and a Multilayer Perceptron for processing clinical data. Using these unimodal architectures as building blocks we propose two feature-level multimodal fusion strategies: 1) extracted features, where the unimodal architectures are trained separately and then fused, and 2) end-to-end, where the unimodal architectures are trained together. We show that integration of neuroimaging information with clinical metadata can potentially improve stroke outcome prediction. Additionally, experimental results indicate that the end-to-end fusion approach proves to be more robust.

Characterisation of Cartilage Damage via Fusing Mid-Infrared, Near-Infrared, and Raman Spectroscopic Data

“…Several studies have explored the implementation of multi-modal data fusion to predict blood pressure, stroke or to identify abdominal cancer [27][28][29]. Different data fusion approaches have been described in the literature [28,30,31]; some are based on data compression by PCA and PLS methods before modelling [32,33].…”

Moving Toward Explainable Decisions of Artificial Intelligence Models for the Prediction of Functional Outcomes of Ischemic Stroke Patients

“…Studies so far suggest little benefit of using imaging for functional outcome prediction of ischemic stroke patients. For example, our CNN on whole-brain volumes of baseline Time-of-flight Magnetic Resonance Angiography (TOF-MRA) had a low predictive performance of 90-days mRS (AUC:0.64) (31). Similarly, a CNN applied to pre-EVT CTA images from the MR CLEAN registry (32) also predicted 90-days mRS with average performance (AUC:0.71) (33).…”

“…Considering the advances in AI that allow for multi-modal learning, a natural way to improve predictive performance is to develop DL-based models that process neuroimaging data together with clinical variables available on admission. We, therefore, developed multi-modal neural networks to jointly process whole volumes of TOF-MRA imaging together with patient demographics and clinical variables at admission for predicting 90-days mRS (Figure 2) (31).…”

Timing the Ischemic Stroke by Multiparametric Quantitative Magnetic Resonance Imaging

“…The benefit of applying CNNs to the entire images rather than a pre-specified VOI is that physiological information, crucial to onset time estimates, is less likely to be overlooked (51). DL also affords integrating imaging and clinical data (e.g., age, blood glucose, stroke severity) termed 'multimodal fusion' (52). As well as learning hierarchies of features, the end-to-end approach can learn intramodal representations, which are not specific to imaging or clinical data but are merged representations of the two data types (52).…”

“…DL also affords integrating imaging and clinical data (e.g., age, blood glucose, stroke severity) termed 'multimodal fusion' (52). As well as learning hierarchies of features, the end-to-end approach can learn intramodal representations, which are not specific to imaging or clinical data but are merged representations of the two data types (52). Multimodal fusion has boosted predictive ability in other stroke research areas, such as predicting short-term outcomes (52), and therefore may also improve MRI-based stroke timing methods (43).…”