Rethinking Explainability as a Dialogue: A Practitioner's Perspective

Abstract

As practitioners increasingly deploy machine learning models in critical domains such as healthcare, finance, and policy, it becomes vital to ensure that domain experts function effectively alongside these models. Explainability is one way to bridge the gap between human decision-makers and machine learning models. However, most of the existing work on explainability focuses on one-off, static explanations like feature importances or rule-lists. These sorts of explanations may not be sufficient for many use cases that require dynamic, continuous discovery from stakeholders that have a range of skills and expertise. In the literature, few works ask decision-makers such as doctors, healthcare professionals, and policymakers about the utility of existing explanations and other desiderata they would like to see in an explanation going forward. In this work, we address this gap and carry out a study where we interview doctors, healthcare professionals, and policymakers about their needs and desires for explanations. Our study indicates that decision-makers would strongly prefer interactive explanations. In particular, they would prefer these interactions to take the form of natural language dialogues. Domain experts wish to treat machine learning models as "another colleague", i.e., one who can be held accountable by asking why they made a particular decision through expressive and accessible natural language interactions. Considering these needs, we outline a set of five principles researchers should follow when designing interactive explanations as a starting place for future work. Further, we show why natural language dialogues satisfy these principles and are a desirable way to build interactive explanations. Next, we provide a design of a dialogue system for explainability, and discuss the risks, trade-offs, and research opportunities of building these systems. Overall, we hope our work serves as a starting place for researchers and engineers to design interactive, natural language dialogue systems for explainability that better serve users' needs.

Explainable Artificial Intelligence for Bayesian Neural Networks: Toward Trustworthy Predictions of Ocean Dynamics

“…Note that throughout the LRP values are scaled by the maximum absolute LRP value for any variable across the ensemble. If the LRP value consistently has the same sign across the quantiles, then we can be confident of the effect this feature has on the output; the piece of information of most interest to practitioners in a recent survey in Lakkaraju et al (2022).…”

“…These XAI methods reveal the extent to which the BNN is fit for purpose for our problem. Moreover, our approach means we can gain a reliable notion of the confidence of the explanation, which has been highlighted as a key area where XAI techniques must improve (Lakkaraju et al, 2022). Applying our XAI techniques to BNNs trained on real-world ocean circulation data in an application designed to understand future climate has the added benefit that we are able to validate and confirm these novel applications of XAI using physical understanding of ocean circulation processes, improving confidence in our BNN predictions.…”

Visual Debates

“…Our proposed debate game uses models' latent knowledge to extract arguments and counterarguments, preserving the arguments' faithfulness to the given model. (Lakkaraju et al 2022) advocates the importance of thinking of explainability as a dialog rather than a fixed attention map or feature attribution. We provide a practical framework in this direction.…”

“…All the methods described so far fall under the category of post-hoc explanations, where the trained model is diagnosed to extract reasons for making a particular decision. Instead, methods such as (Stammer, Schramowski, and Kersting 2021;Lampinen et al 2021;Irving, Christiano, and Amodei 2018) aim to develop an intrinsically transparent and aligned model. We borrow some ideas from (Irving, Christiano, and Amodei 2018) in this work to develop our post-hoc explainability framework.…”

From medical imaging to explainable artificial intelligence

“…What's the next big thing in XAI? According to Lakkaraju et al [277], dialogue could be the most important part of explanation. Create algorithms that not only map out feature importance, but make sure they are capable of providing feedback in natural language.…”