The false hope of current approaches to explainable artificial intelligence in health care

Abstract

The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for decision making for individual patients. In the absence of suitable explainability methods, we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals often associated with explainability, and we caution against having explainability be a requirement for clinically deployed models.

Towards On-Device Dehydration Monitoring Using Machine Learning from Wearable Device’s Data

“…Taking into account the computational limitations of the device, using algorithms with a short training time/low complexity could be of great benefit in dehydration monitoring from data. In terms of interpretability and explainable AI, tree-based models have also proven to achieve the best interpretability characteristics by design [31]. Interpretability is especially required by healthcare applications to provide an informed and clear justification for the model decisions.…”

“…Interpretability is especially required by healthcare applications to provide an informed and clear justification for the model decisions. For other types of black-box models that are not interpretable, explainable AI techniques, such as features relevance and visualizations, are utilized to promote confidence in the model, fairness, and informativeness [31]. Using SHAP [32] to visualize the Shapley values for how the features impact the output of the model is shown in Figure 9.…”

Radiomics in Cardiovascular Disease Imaging: from Pixels to the Heart of the Problem

“…On the other hand, the current technology does not allow for this type of information to be actually available or to realistically expect this in the next few years. Some domain experts have already proposed that our attention should not be focused on "understanding" DL models, but rather on requiring strong validation alone [97]. In any case, a consensus should be reached on the actual requirements of radiomics and ML software prior to their approval for clinical use.…”

Crossing the chasm from model performance to clinical impact: the need to improve implementation and evaluation of AI

“…Limited user adoption—due to lack of clinician trust and model interpretability among many other reasons—has long been cited as a key barrier to clinical impact 2 , 3 . Encouraging providers to thoughtfully incorporate a model’s prediction into their decision and ultimate behavior regarding patient care—particularly in scenarios where predictions by the model and the human diverge—is a challenge with no clear solution yet.…”