Machine Learning Workflow to Explain Black-Box Models for Early Alzheimer’s Disease Classification Evaluated for Multiple Datasets

Abstract

Hard-to-interpret black-box Machine Learning (ML) was often used for early Alzheimer’s Disease (AD) detection. To interpret eXtreme Gradient Boosting (XGBoost), Random Forest (RF), and Support Vector Machine (SVM) black-box models, a workflow based on Shapley values was developed. All models were trained on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset and evaluated for an independent ADNI test set, as well as the external Australian Imaging and Lifestyle flagship study of Ageing (AIBL), and Open Access Series of Imaging Studies (OASIS) datasets. Shapley values were compared to intuitively interpretable Decision Trees (DTs), and Logistic Regression (LR), as well as natural and permutation feature importances. To avoid the reduction of the explanation validity caused by correlated features, forward selection and aspect consolidation were implemented. Some black-box models outperformed DTs and LR. The forward-selected features correspond to brain areas previously associated with AD. Shapley values identified biologically plausible associations with moderate-to-strong correlations with feature importances. The most important RF features to predict AD conversion were the volume of the amygdalae and a cognitive test score. Good cognitive test performances and large brain volumes decreased the AD risk. The models trained using cognitive test scores significantly outperformed brain volumetric models ($$p<0.05$$ p < 0.05 ). Cognitive Normal (CN) vs. AD models were successfully transferred to external datasets. In comparison to previous work, improved performances for ADNI and AIBL were achieved for CN vs. Mild Cognitive Impairment (MCI) classification using brain volumes. The Shapley values and the feature importances showed moderate-to-strong correlations.

Application of Explainable Artificial Intelligence in Alzheimer's Disease Classification: A Systematic Review

“…Hamza et al [121] experimented with neural network models for early AD detection by employing classification approaches utilizing a hybrid dataset from Kaggle and OASIS. Louis et al [154] propose a machine learning workflow to train and interpret different Blackbox models and to compare its performance. All models were trained and evaluated on ADNI, AIBL and OASIS datasets.…”

“…They consequently provide the doctors with an understanding of how and why the model makes judgements. SHAP is used by Ahmed et al [152] and Louise et al [154] to determine the order of informative predictors in test data. ML models and their relationships were also visualised and analysed using SHAP summary plots.…”

“…The ML or DL learning algorithms predict dominant classes more accurately than classes with inadequate samples. Most studies had limited AD samples compared to HC or MCI cohorts [158,116,118,144,150,154,126]. Only a balanced dataset can draw meaningful insights.…”

“…Louis Bloch et al [154] used Shapley values to interpret XGBoost, SVM, and RF BlackBox models using the ADNI dataset. The study considered MRI volume features, cognitive test results, sociodemographic data, and Apolipoprotein alleles.…”

“…In yet another study [154], SHAP was used with other methods to validate the interpretability. Again, a weaker correlation was found between feature rankings of the SHAP values and other models.…”