Meaningful Explanations of Black Box AI Decision Systems

Pedreschi, Dino; Giannotti, Fosca; Guidotti, Riccardo; Monreale, Anna; Ruggieri, Salvatore; Turini, Franco

doi:10.1609/aaai.v33i01.33019780

Cited by 139 publications

(102 citation statements)

References 22 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…In black box models, it can be challenging to determine what is coordinating the visible patterns. Such models are problematic not only for lack of transparency but also for possible biases inherited by the algorithms from clinicians mistakes [53]. This issue is caused based on the human errors and biased sampling of training data as well as the underestimation of the impact of the risk factors underlying behaviour/pattern.…”

Section: Visualisation In Deep Learningmentioning

confidence: 99%

Predicting Type 2 Diabetes Complications and Personalising Patient Using Artificial Intelligence Methodology

Yousefi¹,

Tucker²

2021

Type 2 Diabetes - From Pathophysiology to Cyber Systems

View full text Add to dashboard Cite

The prediction of the onset of different complications of disease, in general, is challenging due to the existence of unmeasured risk factors, imbalanced data, time-varying data due to dynamics, and various interventions to the disease over time. Scholars share a common argument that many Artificial Intelligence techniques that successfully model disease are often in the form of a “black box” where the internal workings and complexities are extremely difficult to understand, both from practitioners’ and patients’ perspective. There is a need for appropriate Artificial Intelligence techniques to build predictive models that not only capture unmeasured effects to improve prediction, but are also transparent in how they model data so that knowledge about disease processes can be extracted and trust in the model can be maintained by clinicians. The proposed strategy builds probabilistic graphical models for prediction with the inclusion of informative hidden variables. These are added in a stepwise manner to improve predictive performance whilst maintaining as simple a model as possible, which is regarded as crucial for the interpretation of the prediction results. This chapter explores this key issue with a specific focus on diabetes data. According to the literature on disease modelling, especially on major diseases such as diabetes, a patient’s mortality often occurs due to the associated complications caused by the disease over time and not the disease itself. This is often patient-specific and will depend on what type of cohort a patient belongs to. Another main focus of this study is patient personalisation via precision medicine by discovering meaningful subgroups of patients which are characterised as phenotypes. These phenotypes are explained further using Bayesian network analysis methods and temporal association rules. Overall, this chapter discussed the earlier research of the chapter’s author. It explores Artificial Intelligence (IDA) techniques for modelling the progression of disease whilst simultaneously stratifying patients and doing so in a transparent manner as possible. To this end, it reviews the current literature on some of the most common Artificial Intelligent (AI) methodologies, including probabilistic modelling, association rule mining, phenotype discovery and latent variable discovery by using diabetes as a case study.

show abstract

Section: Visualisation In Deep Learningmentioning

confidence: 99%

Predicting Type 2 Diabetes Complications and Personalising Patient Using Artificial Intelligence Methodology

Yousefi¹,

Tucker²

2021

Type 2 Diabetes - From Pathophysiology to Cyber Systems

View full text Add to dashboard Cite

show abstract

Section: Xai and Interpretable Models -Current State Of The Artmentioning

confidence: 99%

“…Yet we wish to communicate explanations to a variety of levels of domain expertise: patient, practitioner, healthcare administrators and regulators. Additionally, we set higher standards of statistical rigour before granting our trust to ML derived decisions and explanations [20,21].…”

Section: Introductionmentioning

confidence: 99%

Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences

Hatwell

Gaber

Azad

2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background Computer Aided Diagnostics (CAD) can support medical practitioners to make critical decisions about their patients’ disease conditions. Practitioners require access to the chain of reasoning behind CAD to build trust in the CAD advice and to supplement their own expertise. Yet, CAD systems might be based on black box machine learning models and high dimensional data sources such as electronic health records, magnetic resonance imaging scans, cardiotocograms, etc. These foundations make interpretation and explanation of the CAD advice very challenging. This challenge is recognised throughout the machine learning research community. eXplainable Artificial Intelligence (XAI) is emerging as one of the most important research areas of recent years because it addresses the interpretability and trust concerns of critical decision makers, including those in clinical and medical practice. Methods In this work, we focus on AdaBoost, a black box model that has been widely adopted in the CAD literature. We address the challenge – to explain AdaBoost classification – with a novel algorithm that extracts simple, logical rules from AdaBoost models. Our algorithm, Adaptive-Weighted High Importance Path Snippets (Ada-WHIPS), makes use of AdaBoost’s adaptive classifier weights. Using a novel formulation, Ada-WHIPS uniquely redistributes the weights among individual decision nodes of the internal decision trees of the AdaBoost model. Then, a simple heuristic search of the weighted nodes finds a single rule that dominated the model’s decision. We compare the explanations generated by our novel approach with the state of the art in an experimental study. We evaluate the derived explanations with simple statistical tests of well-known quality measures, precision and coverage, and a novel measure stability that is better suited to the XAI setting. Results Experiments on 9 CAD-related data sets showed that Ada-WHIPS explanations consistently generalise better (mean coverage 15%-68%) than the state of the art while remaining competitive for specificity (mean precision 80%-99%). A very small trade-off in specificity is shown to guard against over-fitting which is a known problem in the state of the art methods. Conclusions The experimental results demonstrate the benefits of using our novel algorithm for explaining CAD AdaBoost classifiers widely found in the literature. Our tightly coupled, AdaBoost-specific approach outperforms model-agnostic explanation methods and should be considered by practitioners looking for an XAI solution for this class of models.

show abstract

“…The prevalence of CRL as interpretable models indicates the importance of logical rules for explainability. Logical rules are intuitive to understand, being the standard language of reasoning [20,36] and are the paradigm that we have adopted in our method. The above mentioned methods are examples of globally interpretable proxy models; they allow the user to infer some understanding of the black box model's overall behaviour.…”

Section: Xai and Interpretable Models -Current State Of The Artmentioning

confidence: 99%

Ada-WHIPS: Explaining AdaBoost Classification with Applications in the Health Sciences

Hatwell

Gaber

Azad

2020

Preprint

View full text Add to dashboard Cite

Background Computer Aided Diagnostics (CAD) can support medical practitioners to make critical decisions about their patients' disease conditions. Practitioners require access to the chain of reasoning behind CAD to build trust in the CAD advice and to supplement their own expertise. Yet, CAD systems might be based on black box machine learning (ML) models and high dimensional data sources (electronic health records, MRI scans, cardiotocograms, etc). These foundations make interpretation and explanation of the CAD advice very challenging. This challenge is recognised throughout the machine learning research community. eXplainable Artificial Intelligence (XAI) is emerging as one of the most important research areas of recent years because it addresses the interpretability and trust concerns of critical decision makers, including those in clinical and medical practice. Methods In this work, we focus on AdaBoost, a black box ML model that has been widely adopted in the CAD literature. We address the challenge -- to explain AdaBoost classification -- with a novel algorithm that extracts simple, logical rules from AdaBoost models. Our algorithm, Adaptive-Weighted High Importance Path Snippets (Ada-WHIPS), makes use of AdaBoost's adaptive classifier weights. Using a novel formulation, Ada-WHIPS uniquely redistributes the weights among individual decision nodes of the internal decision trees (DT) of the AdaBoost model. Then, a simple heuristic search of the weighted nodes finds a single rule that dominated the model's decision. We compare the explanations generated by our novel approach with the state of the art in an experimental study. We evaluate the derived explanations with simple statistical tests of well-known quality measures, precision and coverage, and a novel measure stability that is better suited to the XAI setting.Results Experiments on 9 CAD-related data sets showed that Ada-WHIPS explanations consistently generalise better (mean coverage 15%-68%) than the state of the art while remaining competitive for speciﬁcity (mean precision 80%-99%). A very small trade-oﬀ in speciﬁcity is shown to guard againstover-ﬁtting which is a known problem in the state of the art methods.Conclusions The experimental results demonstrate the beneﬁts of using our novel algorithm for explaining CAD AdaBoost classiﬁers widely found in the literature. Our tightly coupled, AdaBoost-speciﬁc approach outperforms model-agnostic explanation methods and should be considered by practitioners looking for an XAI solution for this class of models.

show abstract

Meaningful Explanations of Black Box AI Decision Systems

Cited by 139 publications

References 22 publications

Predicting Type 2 Diabetes Complications and Personalising Patient Using Artificial Intelligence Methodology

Predicting Type 2 Diabetes Complications and Personalising Patient Using Artificial Intelligence Methodology

Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences

Ada-WHIPS: Explaining AdaBoost Classification with Applications in the Health Sciences

Contact Info

Product

Resources

About