Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review

Abstract

<jats:title>Abstract</jats:title> <jats:sec> <jats:title>Objective</jats:title> <jats:p>To conduct a systematic scoping review of explainable artificial intelligence (XAI) models that use real-world electronic health record data, categorize these techniques according to different biomedical applications, identify gaps of current studies, and suggest future research directions.</jats:p> </jats:sec> <jats:sec> <jats:title>Materials and Methods</jats:title> <jats:p>We searched MEDLINE, IEEE Xplore, and the Association for Computing Machinery (ACM) Digital Library to identify relevant papers published between January 1, 2009 and May 1, 2019. We summarized these studies based on the year of publication, prediction tasks, machine learning algorithm, dataset(s) used to build the models, the scope, category, and evaluation of the XAI methods. We further assessed the reproducibility of the studies in terms of the availability of data and code and discussed open issues and challenges.</jats:p> </jats:sec> <jats:sec> <jats:title>Results</jats:title> <jats:p>Forty-two articles were included in this review. We reported the research trend and most-studied diseases. We grouped XAI methods into 5 categories: knowledge distillation and rule extraction (N = 13), intrinsically interpretable models (N = 9), data dimensionality reduction (N = 8), attention mechanism (N = 7), and feature interaction and importance (N = 5).</jats:p> </jats:sec> <jats:sec> <jats:title>Discussion</jats:title> <jats:p>XAI evaluation is an open issue that requires a deeper focus in the case of medical applications. We also discuss the importance of reproducibility of research work in this field, as well as the challenges and opportunities of XAI from 2 medical professionals’ point of view.</jats:p> </jats:sec> <jats:sec> <jats:title>Conclusion</jats:title> <jats:p>Based on our review, we found that XAI evaluation in medicine has not been adequately and formally practiced. Reproducibility remains a critical concern. Ample opportunities exist to advance XAI research in medicine.</jats:p> </jats:sec>

Can Big Data guide prognosis and clinical decisions in epilepsy?

“…Decisions in such systems can be more optimized, but gaps exist in appropriate explanations in the context of expert medical knowledge on how such decisions are derived. Development of effective explainable methods requires more interdisciplinary cooperation between professionals from different domains, such as machine learning researchers and medical experts, 20 and the effectiveness of the interpretation of explainable CDSSs must be evaluated based on how its interpretation helps human users, which may require human‐in‐the‐loop psychology experiments to measure mental models, task performance, user satisfaction, and appropriate trust 21 …”

Conflicting roles for humans in learning health systems and AI‐enabled healthcare

“…Not all algorithms are inscrutable. 47 Google, for instance, published an evaluation of the model used by its algorithm for assessing retinal images. 48 As LHS develop, policies could be put in place that prohibit black box algorithms.…”

Assessing the Impact of Imputation on the Interpretations of Prediction Models: A Case Study on Mortality Prediction for Patients with Acute Myocardial Infarction

“…For more reading on this we refer the interested audience to a recent systematic review on the explainable AI models using EHR data. 23 For example, in a logistic regression model for binary outcome, the coefficients of the features (predictors) can be readily transformed into odds ratios and can be easily understood as feature importance. Nevertheless, as these coefficients are estimated from the input data, when the data points were replaced with different imputation techniques, the extent to which missing data points are extrapolated has certain impact on the interpretation of these coefficients.…”

“…12 Researchers in the field have taken different approaches to address the interpretability of machine learning models, for instance, feature interaction and importance, attention mechanism, data dimensionality reduction, knowledge distillation and rule extraction. 23 Nevertheless, there are still some fundamental issues that need to be addressed such as fidelity of the post-hoc interpretation methods to the reference model, evaluation of the interpretation methods, and design biases due to focusing on the intuition of researchers rather than real end-users’ (medical professionals in this context) needs. For more reading on this we refer the interested audience to a recent systematic review on the explainable AI models using EHR data.…”

“…For more reading on this we refer the interested audience to a recent systematic review on the explainable AI models using EHR data. 23…”