Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Hase, Peter; Bansal, Mohit

doi:10.18653/v1/2020.acl-main.491

Cited by 149 publications

(132 citation statements)

References 19 publications

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…The first steps have been made in this direction in other research fields. 72,73 Nonetheless, further development of XAI applications in chemistry would greatly benefit from meaningful benchmarking, which will require close collaboration between medicinal chemists and computer scientists.…”

Section: Resultsmentioning

confidence: 99%

Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment

Jiménez-Luna

Škalič

Weskamp

et al. 2021

J. Chem. Inf. Model.

View full text Add to dashboard Cite

Graph neural networks are able to solve certain drug discovery tasks such as molecular property prediction and \textit{de novo} molecule generation. However, these models are considered 'black-box' and 'hard-to-debug'. This study aimed to improve modeling transparency for rational molecular design by applying the integrated gradients explainable artificial intelligence (XAI) approach for graph neural network models. Models were trained for predicting plasma protein binding, cardiac potassium channel inhibition, passive permeability, and cytochrome P450 inhibition. The proposed methodology highlighted molecular features and structural elements that are in agreement with known pharmacophore motifs, correctly identified property cliffs, and provided insights into unspecific ligand-target interactions. The developed XAI approach is fully open-sourced and can be used by practitioners to train new models on other clinically-relevant endpoints. File list (2) download file view on ChemRxiv jimenez2020color.pdf (5.85 MiB) download file view on ChemRxiv molgrad_series.csv (13.15 KiB)

show abstract

Section: Resultsmentioning

confidence: 99%

Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment

Jiménez-Luna

Škalič

Weskamp

et al. 2021

J. Chem. Inf. Model.

View full text Add to dashboard Cite

show abstract

“…Model Interpretability: PROVER follows a significant body of previous work on developing interpretable neural models for NLP tasks to foster explainability. Several approaches have focused on formalizing the notion of interpretability (Rudin, 2019;Doshi-Velez and Kim, 2017;Hase and Bansal, 2020), tweaking features for local model interpretability (Ribeiro et al, 2016(Ribeiro et al, , 2018 and exploring interpretability in latent spaces (Joshi et al, 2018;Samangouei et al, 2018). Our work can be seen as generating explanations in the form of proofs for an NLP task.…”

Section: Related Workmentioning

confidence: 99%

PRover: Proof Generation for Interpretable Reasoning over Rules

Saha

Ghosh

Srivastava

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Recent work by Clark et al. (2020) shows that transformers can act as "soft theorem provers" by answering questions over explicitly provided knowledge in natural language. In our work, we take a step closer to emulating formal theorem provers, by proposing PROVER, an interpretable transformer-based model that jointly answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. During inference, a valid proof, satisfying a set of global constraints is generated. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation, with strong generalization performance. First, PROVER generates proofs with an accuracy of 87%, while retaining or improving performance on the QA task, compared to RuleTakers (up to 6% improvement on zero-shot evaluation). Second, when trained on questions requiring lower depths of reasoning, it generalizes significantly better to higher depths (up to 15% improvement). Third, PROVER obtains near perfect QA accuracy of 98% using only 40% of the training data. However, generating proofs for questions requiring higher depths of reasoning becomes challenging, and the accuracy drops to 65% for "depth 5", indicating significant scope for future work. 1

show abstract

“…LAS scores combine two key mechanisms: they measure simulatability, which reflects how well an observer can use model explanations to predict the model's output, while controlling for explanation leakage, which occurs when explanations directly leak the output. This metric is inspired by prior work on model interpretability (Doshi-Velez and Kim, 2017;Hase and Bansal, 2020), but to date no simulatability analysis has been carried out for NL explanations. We automate our evaluation by using a pretrained language model as the observer, serving as a proxy for a human.…”

Section: (Details In Section 3)mentioning

confidence: 99%

Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

Hase¹,

Zhang²,

Xie³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

View full text Add to dashboard Cite

Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores. 1

show abstract

Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Cited by 149 publications

References 19 publications

Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment

Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment

PRover: Proof Generation for Interpretable Reasoning over Rules

Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

Contact Info

Product

Resources

About