Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice

Background: Accessible datasets are of fundamental importance to the advancement of Alzheimer’s disease (AD) research. The AddNeuroMed consortium conducted a longitudinal observational cohort study with the aim to discover AD biomarkers. During this study, a broad selection of data modalities was measured including clinical assessments, magnetic resonance imaging, genotyping, transcriptomic profiling, and blood plasma proteomics. Some of the collected data were shared with third-party researchers. However, this data was incomplete, erroneous, and lacking in interoperability. Objective: To provide the research community with an accessible, multimodal, patient-level AD cohort dataset. Methods: We systematically addressed several limitations of the originally shared data and provided additional unreleased data to enhance the patient-level dataset. Results: In this work, we publish and describe ANMerge, a new version of the AddNeuroMed dataset. ANMerge includes multimodal data from 1,702 study participants and is accessible to the research community via a centralized portal. Conclusion: ANMerge is an information rich patient-level data resource that can serve as a discovery and validation cohort for data-driven AD research, such as, for example, machine learning and artificial intelligence approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

ANMerge: A Comprehensive and Accessible Alzheimer’s Disease Patient-Level Dataset

Birkenbihl

Westwood

Shi

et al. 2021

JAD

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

“…To validate discoveries made in ADNI on other datasets, a high overlap in measured variables is a prerequisite. Previously, we could demonstrate that despite evident differences to ADNI, ANMerge is a viable validation dataset(Birkenbihl et al, 2020).Recently, more studies such as PREVENT-AD (Tremblay-Mercier et al, 2014) and EPAD (Solomon et al, 2018) joined the ranks of ADNI, DIAN (Morris et al, 2012) and others by making their data accessible to third party researchers. The currently running Deep Frequent Phenotype Study (Lawson et al, 2017) already emphasized that collected data will be published.…”

mentioning

confidence: 92%

ANMerge: A comprehensive and accessible Alzheimer’s disease patient-level dataset

Birkenbihl

Westwood

Shi

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Accessible datasets are of fundamental importance to the advancement of Alzheimer's disease (AD) research. The AddNeuroMed consortium conducted a longitudinal observational cohort study with the aim to discover AD biomarkers. During this study, a broad selection of data modalities was measured including clinical assessments, magnetic resonance imaging, genotyping, transcriptomic profiling and blood plasma proteomics. Some of the collected data were shared with third-party researchers. However, this data was incomplete, erroneous and lacking in interoperability. Methods: We systematically addressed several limitations of the originally shared data and provide additional unreleased data to enhance the patient-level dataset. Results: In this work, we publish and describe ANMerge, a new version of the AddNeuroMed dataset. ANMerge includes multimodal data from 1702 study participants and is accessible to the research community via a centralized portal. Conclusions: ANMerge is an information rich patient-level data resource that can serve as a discovery and validation cohort for data-driven AD research, such as for example machine learning and artificial intelligence approaches. ANMerge can be downloaded here: https://doi.org/10.7303/syn22252881

show abstract

“…A summary [14] of the properties of AD cohorts shows that AD cohorts differ with regards to study location, study size, recruitment criteria, diagnosis method and biomarkers. A comparison [15] of two AD cohorts, the Alzheimer's Disease Neuroimaging Initiative (ADNI) [16] and AddNeuroMed [17] found major demographical differences between the AD cohorts using a statistical matching procedure. It was found that the demographics of the AddNeuroMed data set were less diverse than the subjects of the ADNI data set.…”

Section: External Validationmentioning

confidence: 99%

Data Analysis With Shapley Values For Automatic Subject Selection in Alzheimer's Disease Data Sets Using Interpretable Machine Learning

Bloch

Friedrich

2021

Preprint

View full text Add to dashboard Cite

Background: The prediction of whether Mild Cognitive Impaired (MCI) subjects will prospectively develop Alzheimer's Disease (AD) is important for the recruitment and monitoring of subjects for therapy studies. Machine Learning (ML) is suitable to improve early AD prediction. The etiology of AD is heterogeneous, which leads to noisy data sets. Additional noise is introduced by multicentric study designs and varying acquisition protocols. This article examines whether an automatic and fair data valuation method based on Shapley values can identify subjects with noisy data. Methods: An ML-workow was developed and trained for a subset of the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort. The validation was executed for an independent ADNI test data set and for the Australian Imaging, Biomarker and Lifestyle Flagship Study of Ageing (AIBL) cohort. The workow included volumetric Magnetic Resonance Imaging (MRI) feature extraction, subject sample selection using data Shapley, Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) for model training and Kernel SHapley Additive exPlanations (SHAP) values for model interpretation. This model interpretation enables clinically relevant explanation of individual predictions. Results: The XGBoost models which excluded 116 of the 467 subjects from the training data set based on their Logistic Regression (LR) data Shapley values outperformed the models which were trained on the entire training data set and which reached a mean classification accuracy of 58.54 % by 14.13 % (8.27 percentage points) on the independent ADNI test data set. The XGBoost models, which were trained on the entire training data set reached a mean accuracy of 60.35 % for the AIBL data set. An improvement of 24.86 % (15.00 percentage points) could be reached for the XGBoost models if those 72 subjects with the smallest RF data Shapley values were excluded from the training data set. Conclusion: The data Shapley method was able to improve the classification accuracies for the test data sets. Noisy data was associated with the number of ApoEϵ4 alleles and volumetric MRI measurements. Kernel SHAP showed that the black-box models learned biologically plausible associations.

show abstract

Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice

Cited by 38 publications

References 40 publications

ANMerge: A Comprehensive and Accessible Alzheimer’s Disease Patient-Level Dataset

ANMerge: A Comprehensive and Accessible Alzheimer’s Disease Patient-Level Dataset

ANMerge: A comprehensive and accessible Alzheimer’s disease patient-level dataset

Data Analysis With Shapley Values For Automatic Subject Selection in Alzheimer's Disease Data Sets Using Interpretable Machine Learning

Contact Info

Product

Resources

About