A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

Mohseni, Sina; Zarei, Niloofar; Ragan, Eric D.

doi:10.48550/arxiv.1811.11839

Cited by 27 publications

(55 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Global explanations try to provide an overview of how a model generates its outputs [2,32]. Many explainable systems designed for data experts have focused on visualizing models as global explanations [57]. For instance, Hohman et al [32] built an interactive visual system to summarize and visualize deep-learning models and show how much each layer and what features were used to make predictions.…”

Section: Explanation Meaningfulness and Veracitymentioning

confidence: 99%

“…Effective evaluation of XAI systems is challenging because it must not only assess how the addition of explanations can improve user understanding and trust in the system but also whether improvements in understanding allows users to work more efficiently [57]. Human evaluation should also aim to understand which aspects or types of explanations aid human understanding-especially when several types of explanations are provided to the user.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

Nourani,

Roy,

Rahman

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Explainable machine learning and artificial intelligence models have been used to justify a model's decisionmaking process. This added transparency aims to help improve user performance and understanding of the underlying model. However, in practice, explainable systems face many open questions and challenges. Specifically, designers might reduce the complexity of deep learning models in order to provide interpretability. The explanations generated by these simplified models, however, might not accurately justify and be truthful to the model. This can further add confusion to the users as they might not find the explanations meaningful with respect to the model predictions. Understanding how these explanations affect user behavior is an ongoing challenge. In this paper, we explore how explanation veracity affects user performance and agreement in intelligent systems. Through a controlled user study with an explainable activity recognition system, we compare variations in explanation veracity for a video review and querying task. The results suggest that low veracity explanations significantly decrease user performance and agreement compared to both accurate explanations and a system without explanations. These findings demonstrate the importance of accurate and understandable explanations and caution that poor explanations can sometimes be worse than no explanations with respect to their effect on user performance and reliance on an AI system. CCS Concepts: • Human-centered computing → Empirical studies in HCI; User studies; • Computing methodologies → Machine learning approaches.

show abstract

Section: Explanation Meaningfulness and Veracitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

Nourani,

Roy,

Rahman

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…etc., which toughens the comprehensive analysis. These bottlenecks undermine not only any qualitative assessment but also quantitative metrics requiring user interactions [29]. Furthermore, expressing the quantitative metrics in user-understandable terminologies [28] is fundamental to achieve interpretability [30], [31].…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, expressing the quantitative metrics in user-understandable terminologies [28] is fundamental to achieve interpretability [30], [31]. To this end, the most popular quantitative metric, explainer fidelity [26], [29], [32]- [35], is not satisfactory. Moreover, explainers intrinsically maintain high-fidelity, e.g., GNNEXPLAINER [27] produces an explanation to match the GNN's prediction on the original graph.…”

Section: Introductionmentioning

confidence: 99%

Quantifying Explainers of Graph Neural Networks in Computational Pathology

Jaume¹,

Pati²,

Bozorgtabar³

et al. 2020

Preprint

View full text Add to dashboard Cite

Explainability of deep learning methods is imperative to facilitate their clinical adoption in digital pathology. However, popular deep learning methods and explainability techniques (explainers) based on pixel-wise processing disregard biological entities' notion, thus complicating comprehension by pathologists. In this work, we address this by adopting biological entity-based graph processing and graph explainers enabling explanations accessible to pathologists. In this context, a major challenge becomes to discern meaningful explainers, particularly in a standardized and quantifiable fashion. To this end, we propose herein a set of novel quantitative metrics based on statistics of class separability using pathologically measurable concepts to characterize graph explainers. We employ the proposed metrics to evaluate three types of graph explainers, namely the layer-wise relevance propagation, gradient-based saliency, and graph pruning approaches, to explain Cell-Graph representations for Breast Cancer Subtyping. The proposed metrics are also applicable in other domains by using domainspecific intuitive concepts. We validate the qualitative and quantitative findings on the BRACS dataset, a large cohort of breast cancer RoIs, by expert pathologists.

show abstract

“…

…”

mentioning

confidence: 99%

A Multistakeholder Approach Towards Evaluating AI Transparency Mechanisms

Lucic,

Srikumar,

Bhatt

et al. 2021

Preprint

View full text Add to dashboard Cite

Given that there are a variety of stakeholders involved in, and affected by, decisions from machine learning (ML) models, it is important to consider that different stakeholders have different transparency needs [14]. Previous work found that the majority of deployed transparency mechanisms primarily serve technical stakeholders [2]. In our work, we want to investigate how well transparency mechanisms might work in practice for a more diverse set of stakeholders by conducting a large-scale, mixed-methods user study across a range of organizations, within a particular industry such as health care, criminal justice, or content moderation. In this paper, we outline the setup for our study.

show abstract

A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

Cited by 27 publications

References 0 publications

Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

Quantifying Explainers of Graph Neural Networks in Computational Pathology

A Multistakeholder Approach Towards Evaluating AI Transparency Mechanisms

Contact Info

Product

Resources

About