Multi-Modal Deep Learning Models for Alzheimer's Disease Prediction Using MRI and EHR

Prabhu, Sathvik S.; Berkebile, John A.; Rajagopalan, Neha; Yao, Renjie; Shi, Wenqi; Giuste, Felipe; Zhong, Yishan; Sun, Jimin; Wang, May D.

doi:10.1109/bibe55377.2022.00044

Cited by 6 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It trained a deep auto-encoder to extract features from EHR data, and ResNet and 3D U-Net for MRI image data, followed by an entropy-based weighted sum classification method to combine the results from each modality and generate a final prediction. 20 The traditional approach to clustering involves grouping patients based on their static or longitudinal covariates in an unsupervised manner. 21 However, this approach does not take the observed outcomes of the patients into consideration, such as the onset of comorbidities, and adverse events.…”

Section: Introductionmentioning

confidence: 99%

Identification of Outcome-Oriented Progression Subtypes from Mild Cognitive Impairment to Alzheimer’s Disease Using Electronic Health Records

Yin

et al. 2023

Preprint

View full text Add to dashboard Cite

Alzheimer's disease (AD) is a complex heterogeneous neurodegenerative disease that requires an in-depth understanding of its progression pathways and contributing factors to develop effective risk stratification and prevention strategies. In this study, we proposed an outcome-oriented model to identify progression pathways from mild cognitive impairment (MCI) to AD using electronic health records (EHRs) from the OneFlorida+ Clinical Research Consortium. To achieve this, we employed the long short-term memory (LSTM) network to extract relevant information from the sequential records of each patient. The hierarchical agglomerative clustering was then applied to the learned representation to group patients based on their progression subtypes. Our approach identified multiple progression pathways, each of which represented distinct patterns of disease progression from MCI to AD. These pathways can serve as a valuable resource for researchers to understand the factors influencing AD progression and to develop personalized interventions to delay or prevent the onset of the disease.

show abstract

Section: Introductionmentioning

confidence: 99%

Identification of Outcome-Oriented Progression Subtypes from Mild Cognitive Impairment to Alzheimer’s Disease Using Electronic Health Records

Yin

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Ongoing efforts are being made to overcome the limitations of image-only models by fusing medical images with EHR data [18]. In prior research, Prabhu et al applied MRI images combined with EHR data to Multi-Modal Deep Learning Models for the classification of Alzheimer's Disease [19]. Jabbour et al diagnosed acute respiratory failure (ARF) by applying chest X-rays and EHR data to CNN and ANN models, respectively [20].…”

Section: Introductionmentioning

confidence: 99%

Toward Better Ear Disease Diagnosis: A Multi-Modal Multi-Fusion Model Using Endoscopic Images of the Tympanic Membrane and Pure-Tone Audiometry

Kim,

Kim

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Chronic otitis media is characterized by recurrent infections, leading to serious complications, such as meningitis, facial palsy, and skull base osteomyelitis. Therefore, active treatment based on early diagnosis is essential. This study developed a multi-modal multi-fusion (MMMF) model that automatically diagnoses ear diseases by applying endoscopic images of the tympanic membrane (TM) and pure-tone audiometry (PTA) data to a deep learning model. The primary aim of the proposed MMMF model is adding "normal with hearing loss" as a category, and improving the diagnostic accuracy of the conventional four ear diseases: normal, TM perforation, retraction, and cholesteatoma. To this end, the MMMF model was trained on 1,480 endoscopic images of the TM and PTA data to distinguish five ear disease states: normal, TM perforation, retraction, cholesteatoma, and normal (hearing loss). It employs a feature fusion strategy of crossattention, concatenation, and gated multi-modal units in a multi-modal architecture encompassing a convolutional neural network (CNN) and multi-layer perceptron. We expanded the classification capability to include an additional category, normal (hearing loss), thereby enhancing the diagnostic performance of extant ear disease classification. The MMMF model demonstrated superior performance when implemented with EfficientNet-B7, achieving 92.9% accuracy and 90.9% recall, thereby outpacing the existing feature fusion methods. In addition, five-fold cross-validation experiments were conducted, in which the model consistently demonstrated robust performance when endoscopic images of the TM and PTA data were applied to the deep learning model across all datasets. The proposed MMMF model is the first to include a category of normal ear disease state with hearing loss. The developed model demonstrated superior performance compared to existing CNN models and feature fusion methods. Consequently, this study substantiates the utility of simultaneously applying PTA data and endoscopic images of the TM for the automated diagnosis of ear diseases in clinical settings and validates the usefulness of the multi-fusion method.

show abstract

Alzheimer’s disease diagnosis from single and multimodal data using machine and deep learning models: Achievements and future directions

Elazab,

Wang,

Abdelaziz

et al. 2024

Expert Systems with Applications

View full text Add to dashboard Cite

Multi-Modal Deep Learning Models for Alzheimer's Disease Prediction Using MRI and EHR

Cited by 6 publications

References 19 publications

Identification of Outcome-Oriented Progression Subtypes from Mild Cognitive Impairment to Alzheimer’s Disease Using Electronic Health Records

Identification of Outcome-Oriented Progression Subtypes from Mild Cognitive Impairment to Alzheimer’s Disease Using Electronic Health Records

Toward Better Ear Disease Diagnosis: A Multi-Modal Multi-Fusion Model Using Endoscopic Images of the Tympanic Membrane and Pure-Tone Audiometry

Alzheimer’s disease diagnosis from single and multimodal data using machine and deep learning models: Achievements and future directions

Contact Info

Product

Resources

About