Artificial intelligence-based methods for fusion of electronic health records and imaging data

Abstract

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when using multimodal data is how to fuse them—a field of growing interest among researchers. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that fulfilled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020 to 2021. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective, conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for researchers interested in knowing the current state of knowledge within this research field.

Society for birth defects research and prevention's multidisciplinary research needs workshop 2022: A call to action

“…Application of AI/ML to the prediction of mechanisms has been utilized (Davenport & Kalakota, 2019; Vamathevan et al, 2019) although applications to regulatory decision making are not straightforward due to the challenges integrating data across different biological scales (e.g., molecular, cellular, tissue, organismal). Several examples of clinical diagnosis were discussed. However, it was evident that the use of AI/ML was most readily demonstrated for data mining and diagnostic imaging (Anklam et al, 2022; Mohsen et al, 2022). The translatable application for of AI/ML for risk factor analysis for prognosis and pattern recognition were less frequently noted. As a collection of structured and unstructured data from many different sources, mobilizing big data for identifying data streams that offer the characteristics of volume, value, velocity, variety, and veracity has an important role for AI/ML.…”

“…Several examples of clinical diagnosis were discussed. However, it was evident that the use of AI/ML was most readily demonstrated for data mining and diagnostic imaging (Anklam et al, 2022;Mohsen et al, 2022). The translatable application for of AI/ML for risk factor analysis for prognosis and pattern recognition were less frequently noted.…”

Revolutionizing nursing education and care: The role of artificial intelligence in nursing

“…Integrating imaging data with specific laboratory test results and demographic data leads to improved outcomes. 8 As EHR data is complex and contains diagnosis, scans, laboratory test results, administrative notes doctor's signature it is difficult in working with EHR data. Because of this complexity, researchers use the combination of AI and EHR.…”

“…Thanks to the advances in AI and machine learning (ML) models, multimodal data fusion with different features can be achieved. Integrating imaging data with specific laboratory test results and demographic data leads to improved outcomes 8 …”

Comparison of two causal inference methods: linear regression and matching score and introduction of causal forest

“…Traditional artificial intelligence (AI)is fundamentally based on correlation analysis and lacks a deep understanding of causality. The correlation analysis can only imply that these things are related, not explain the internal logic [1]. Emerging causal inference methods enable us to infer causal structures from data, select interventions effectively to check out putative causality and make better decisions by using knowledge of causal structures.…”