Deep Learning Classification of Cardiomegaly Using Combined Imaging and Non-imaging ICU Data

Abstract

In this paper, we investigate the classification of cardiomegaly using multimodal data, combining imaging data from chest radiography with routinely collected Intensive Care Unit (ICU) data comprising vital sign values, laboratory measurements, and admission metadata. In practice a clinician would assess for the presence of cardiomegaly using a synthesis of multiple sources of data, however, prior machine learning approaches to this task have focused on chest radiographs only. We show that non-imaging ICU data can be used for cardiomegaly classification and propose a novel multimodal network trained simultaneously on both chest radiographs and ICU data. We compare the predictive power of both single-mode approaches with the joint network. We use a subset of data from the publicly available MIMIC-CXR and MIMIC-IV datasets, which contain both chest radiographs and non-imaging ICU data for the same patients. The approach from non-imaging ICU data alone achieves an AUC of 0.684 and the standard chest radiography approach an AUC of 0.840. Our joint model achieves an AUC of 0.880. We conclude that non-imaging ICU data have predictive value for cardiomegaly, and that combining chest radiographs with non-imaging ICU data has the potential to improve model performance for the same subset of patients, with further work required to demonstrate a significant improvement.

Artificial intelligence-based methods for fusion of electronic health records and imaging data

“…As presented in Table 2, approximately two-thirds of the studies were journal articles (n= 23, ∼ 68%) 12-15, 17, 19, 25, 32-46 , whereas 11 studies were conference proceedings (∼ 32%) 16,[47][48][49][50][51][52][53][54][55][56] . Most of the studies were published between 2020 and 2022 (n = 22, ∼ 65%).…”

“…Then, they concatenated the learned features of the two modalities before feeding them into a stacked K-nearest neighbor (KNN) attention pooling layer. Grant et al 55 used a Residual Network (ResNet50) architecture to extract relevant features from the imaging modality and fully connected NN to process the non-imaging data. They directly concatenated the learned feature representation of the imaging and non-imaging data and fed them into two fully connected networks.…”

“…Then, they jointly learned the non-linear correlations among all modalities using fully connected NN. Out of 10 joint fusion studies, seven studies evaluated their fusion models' performance against that of a single modality and reported a performance improvement when fusion was used 17,39,46,47,49,53,55 .…”

“…The most common applied clinical outcome in the included studies was the diagnosis, reported in 20 (∼ 59%) studies. In these studies, EHRs were combined with medical imaging to diagnose a spectrum of diseases including neurological disorders (n = 9) 4, 13-15, 32, 37, 42, 49, 50 , psychiatric disorders (n = 2) 33,36 , CVD (n =3) [54][55][56] , Cancer (n = 2) 16,55 , and four studies for other different diseases 18,19,40,53 . Specifically, most of the studies that focused on detecting neurological diseases were for AD (n = 4) [13][14][15]49 , and MCI (n = 4) 37,42,50,51 .…”

“…Moreover, two studies implemented multimodality early fusion to diagnose different cancer diseases 16,55 . Yan et al 16 fused pathological images and structured data extracted from EHRs to classify malignant and benign breast cancer.…”