Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Vasconcelos, M. Helena; Jörke, Matthew; Grunde-McLaughlin, Madeleine; Bernstein, Michael A.; Krishna, Ranjay

doi:10.48550/arxiv.2212.06823

Cited by 8 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In summary, by removing the limitation of fixed sampling in Study 1, the results in Study 2 demonstrate the promise of selective explanations in improving performance and reducing over-reliance. These results are especially exciting given the growing concerns about the XAI pitfall leading to over-reliance when the AI is wrong [6,91]. We further reflect on this result and its implications in Discussions.…”

Section: Effect Of Selective Explanations On Random Samplesmentioning

confidence: 62%

“…Research has attributed this phenomenon to a lack of cognitive engagement with AI explanations [11,35,52,62]: when people lack either the motivation or ability to carefully analyze and reason about explanations, they make a quick heuristic judgment, which tends to superficially associate being explainable to being trustworthy [28,61]. A recent CSCW work by Vasconcelos et al [91] further calls out that this lack of cognitive engagement will persist if XAI techniques remain hard to use, as people strategically choose to engage with explanations or simply defer to AI by weighing the cognitive costs.…”

Section: Explainable Ai and Its Pitfallsmentioning

confidence: 99%

“…These AI explanations often lack intuitiveness and demand significant effort for people to process and understand them [12,34,55]. They have been found to risk impairing task performance [6,40,41,57,80], efficiency [1,17,34,55,97], and user satisfaction [24,55,67], and may ultimately lead to disengagement from users to harness any benefits that they can gain from explanations [12,35,91]. This difficulty to use current XAI techniques can be attributed to their lack of compatibility with how humans produce and consume explanations, as criticized by many drawing from the social sciences literature [62,73,94].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Selective Explanations: Leveraging Human Input to Align Explainable AI

Lai¹,

Zhang²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

While a vast collection of explainable AI (XAI) algorithms have been developed in recent years, they are often criticized for significant gaps with how humans produce and consume explanations. As a result, current XAI techniques are often found to be hard to use and lack effectiveness. In this work, we attempt to close these gaps by making AI explanations selective-a fundamental property of human explanations-by selectively presenting a subset from a large set of model reasons based on what aligns with the recipient's preferences. We propose a general framework for generating selective explanations by leveraging human input on a small sample.This framework opens up a rich design space that accounts for different selectivity goals, types of input, and more. As a showcase, we use a decision-support task to explore selective explanations based on what the decision-maker would consider relevant to the decision task. We conducted two experimental studies to examine three out of a broader possible set of paradigms based on our proposed framework: in Study 1, we ask the participants to provide their own input to generate selective explanations, with either open-ended or critique-based input. In Study 2, we show participants selective explanations based on input from a panel of similar users (annotators). Our experiments demonstrate the promise of selective explanations in reducing over-reliance on AI and improving decision outcomes and subjective perceptions of the AI, but also paint a nuanced picture that attributes some of these positive effects to the opportunity to provide one's own input to augment AI explanations. Overall, our work proposes a novel XAI framework inspired by human communication behaviors and demonstrates its potentials to encourage future work to better align AI explanations with human production and consumption of explanations. CCS Concepts: • Human-centered computing → Collaborative and social computing; • Computing methodologies → Artificial intelligence; • Applied computing → Law, social and behavioral sciences.

show abstract

Section: Effect Of Selective Explanations On Random Samplesmentioning

confidence: 62%

Section: Explainable Ai and Its Pitfallsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Selective Explanations: Leveraging Human Input to Align Explainable AI

Lai¹,

Zhang²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, it remains to be demonstrated that the observed benefits (task completion time, targeted edits, preference), translate to increased oversight and decreased automation bias. Nevertheless, we are encouraged that past work has shown that reducing the effort needed to interpret model explanations, or expressions of uncertainty, can increase the likelihood of people overriding AI-induced errors [51]. We hope to explore these questions in immediate future work.…”

Section: Impact On Automation Biasmentioning

confidence: 99%

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

Vasconcelos¹,

Bansal²,

Fourney³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large-scale generative models enabled the development of AI-powered code completion tools to assist programmers in writing code. However, much like other AI-powered tools, AI-powered code completions are not always accurate, potentially introducing bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers identify potential errors is to highlight uncertain tokens. However, there have been no empirical studies exploring the effectiveness of this technique-nor investigating the different and not-yet-agreed-upon notions of uncertainty in the context of generative models. We explore the question of whether conveying information about uncertainty enables programmers to more quickly and accurately produce code when collaborating with an AI-powered code completion tool, and if so, what measure of uncertainty best fits programmers' needs. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools, and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming. This work contributes to building an understanding of what uncertainty means for generative models and how to convey it effectively.

show abstract

“…For humans that are assisted by AI, it is therefore essential to be able to identify strengths and weaknesses of the AI system (i.e., in which cases it is correct and in which wrong, see [9]). In this setting, latest research distinguishes three cases of reliance behavior: (i) relying on AI recommendations in too few cases (i.e., under-reliance, see [10,11], e.g., by underestimating AI performance), (ii) relying on AI recommendations in too many cases (i.e., over-reliance, see [1,12,13], e.g., by overestimating AI performance), and (iii) relying appropriately on AI recommendations (i.e., adhering to AI recommendations when correct and overriding when wrong, see [5,9,14]). Thus far, research has identified many scenarios in which underreliance or over-reliance results in reduced decision-making performance (e.g., [12,15]).…”

Section: Introductionmentioning

confidence: 99%

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

Schoeffer

Jakubik

Voessing

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

In AI-assisted decision-making, a central promise of putting a human in the loop is that they should be able to complement the AI system by adhering to its correct and overriding its mistaken recommendations. In practice, however, we often see that humans tend to over- or under-rely on AI recommendations, meaning that they either adhere to wrong or override correct recommendations. Such reliance behavior is detrimental to decision-making accuracy. In this work, we articulate and analyze the interdependence between reliance behavior and accuracy in AI-assisted decision-making, which has been largely neglected in prior work. We also propose a visual framework to make this interdependence more tangible. This framework helps us interpret and compare empirical findings, as well as obtain a nuanced understanding of the effects of interventions (e.g., explanations) in AI-assisted decision-making. Finally, we infer several interesting properties from the framework: (i) when humans under-rely on AI recommendations, there may be no possibility for them to complement the AI in terms of decision-making accuracy; (ii) when humans cannot discern correct and wrong AI recommendations, no such improvement can be expected either; (iii) interventions may lead to an increase in decision-making accuracy that is solely driven by an increase in humans’ adherence to AI recommendations, without any ability to discern correct and wrong. Our work emphasizes the importance of measuring and reporting both effects on accuracy and reliance behavior when empirically assessing interventions.

show abstract

Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Cited by 8 publications

References 0 publications

Selective Explanations: Leveraging Human Input to Align Explainable AI

Selective Explanations: Leveraging Human Input to Align Explainable AI

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

Contact Info

Product

Resources

About