How Cognitive Biases Affect XAI-assisted Decision-making

Abstract

The field of eXplainable Artificial Intelligence (XAI) aims to bring transparency to complex AI systems. Although it is usually considered an essentially technical field, effort has been made recently to better understand users' human explanation methods and cognitive constraints. Despite these advances, the community lacks a general vision of what and how cognitive biases affect explainability systems. To address this gap, we present a heuristic map which matches human cognitive biases with explainability techniques from the XAI literature, structured around XAI-aided decision-making. We identify four main ways cognitive biases affect or are affected by XAI systems: 1) cognitive biases affect how XAI methods are designed, 2) they can distort how XAI techniques are evaluated in user studies, 3) some cognitive biases can be successfully mitigated by XAI techniques, and, on the contrary, 4) some cognitive biases can be exacerbated by XAI techniques. We construct this heuristic map through the systematic review of 37 papers-drawn from a corpus of 285-that reveal cognitive biases in XAI systems, including the explainability method and the user and task types in which they arise. We use the findings from our review to structure directions for future XAI systems to better align with people's cognitive processes.

Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making

“…However, empirical research reveals several limitations within the existing AI-assisted decision-making framework, wherein AI acts primarily as a recommender. One notable issue is that individuals, when passively receiving AI suggestions, seldom engage in analytical thinking [3,7,38]. Furthermore, people frequently inappropriately rely on the AI's recommendations (such as overreliance and under-reliance) [8,30,33,46] and the mere provision of AI explanations can, paradoxically, exacerbate overreliance [2,37].…”

Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making

“…Here, we focus on participants' experiences and perceptions in different conditions. Specifically, referring to and adapted from related works, we investigate the following subjective measures as 7-point Likert scale questions in the exit survey (1: Strongly Disagree, 7: Strongly Agree): (1) Trust in AI [14,39]; (2) Confidence in the decision-making process [52,82]; (3) Perceived complexity of the system [14]; (4) Mental demand [14,39,42,52]; (5) Perceived autonomy [44]; (6) Satisfaction [39]; (7) Future use [12]; (8) Trust in the estimation of human-AI CL; (9) Perceived usefulness of estimation of human-AI CL [56]; (10) Perceived helpfulness to decide when to trust the AI [56]); and (11) Acceptance of estimation of their CL. Besides these questions, we also asked participants open-ended questions about how they used and perceived the communicated human-AI CL, and how their decision-making processes were affected by different interface designs.…”

“…Our work proposes leaving the computational probability estimation/comparison task to the system and calibrating human trust by automatically adapting the decision-making process/interface. This can counter possible human cognitive biases [8,94] and avoid making people directly deal with probabilities. Future work could explore two other directions.…”

Will You Accept the AI Recommendation? Predicting Human Behavior in AI-Assisted Decision Making

“…However, recent research has unveiled a potential drawback of providing AI explanations: the risk of increased over-reliance on AI systems when AI provides incorrect suggestions [2,65,84,90]. This phenomenon is attributed to a lack of cognitive engagement with AI explanations, as individuals may opt for quick heuristic judgments, associating explainability with trustworthiness when they lack the motivation or ability for in-depth analysis [3,6].…”

“…This yielded a necessary sample size of 232 participants. After obtaining institutional IRB approval, we recruited participants from Prolific 3…”