Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Abstract

Prior work has identified a resilient phenomenon that threatens the performance of human-AI decision-making teams: overreliance, when people agree with an AI, even when it is incorrect. Surprisingly, overreliance does not reduce when the AI produces explanations for its predictions, compared to only providing predictions. Some have argued that overreliance results from cognitive biases or uncalibrated trust, attributing overreliance to an inevitability of human cognition. By contrast, our paper argues that people strategically choose whether or not to engage with an AI explanation, demonstrating empirically that there are scenarios where AI explanations reduce overreliance. To achieve this, we formalize this strategic choice in a cost-benefit framework, where the costs and benefits of engaging with the task are weighed against the costs and benefits of relying on the AI. We manipulate the costs and benefits in a maze task, where participants collaborate with a simulated AI to find the exit of a maze. Through 5 studies (N = 731), we find that costs such as task difficulty (Study 1), explanation difficulty (Study 2, 3), and benefits such as monetary compensation (Study 4) affect overreliance. Finally, Study 5 adapts the Cognitive Effort Discounting paradigm to quantify the utility of different explanations, providing further support for our framework. Our results suggest that some of the null effects found in literature could be due in part to the explanation not sufficiently reducing the costs of verifying the AI's prediction.

Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking

“…At a more fundamental level, our study raises some caution: while developers and researchers are investing in improving AI models and how LLMs can provide context-specific guidance, users may not always perceive these enhancements as substantial improvements in accuracy and relevance. Recent literature has already raised concerns about users' over-reliance on AI systems, such as in the context of AI-based maze-solving tasks [49]. Although the landscape of user behaviors and mental models is more multifaceted with LLMs, our study demonstrates a similar phenomena of overtrust with LLMs.…”

Future Me: Promoting a future-oriented outlook through conversation with our future self.

“…Designers and researchers who use a chatbot to enact certain important roles for the user or the user themselves, must be aware of potential risks of these practices, including inferring identity, the breach of privacy, and how these practices may affect the person and their social environment. For example: Individuals may rely on the judgment on the chatbot [53] neglecting their own feelings or evidence of their surroundings. Therefore, designers of such interventions need to help users to fully understand the limitations of the proposed intervention and be aware of potential biases induced by the underlying technology.…”

Language-based game theory in the age of artificial intelligence

“…However, these potential benefits come with critical challenges. Previous literature has identified several potential issues, such as accountability, especially when AI-guided decisions lead to adverse outcomes, and the risk of overreliance, which could erode human judgement and decision-making abilities [31][32][33]. In this article, we shift focus to an often-overlooked aspect: the linguistic description of the decision context.…”