On Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making

Schoeffer, Jakob; De‐Arteaga, Maria; Kuehl, Niklas

doi:10.48550/arxiv.2209.11812

Cited by 4 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extending it to account for varying AI accuracy would involve a 3-dimensional visual with a third axis on Acc AI . Finally, we might think of cases where the metric of decision-making performance is not accuracy but, for instance, fairness [5].…”

Section: Discussionmentioning

confidence: 99%

“…In order to complement the AI system, the human would have to adhere to its recommendations if and only if these recommendations are correct and override them otherwise. Empirical studies have shown, however, that humans are often not able to achieve this type of appropriate reliance 2 [1,4,5,6]. Instead, we often observe that humans either over-or under-rely on AI recommendations, or simply cannot calibrate their reliance.…”

Section: Introductionmentioning

confidence: 95%

“…For humans that are assisted by AI, it is therefore essential to be able to identify strengths and weaknesses of the AI system (i.e., in which cases it is correct and in which wrong, see [9]). In this setting, latest research distinguishes three cases of reliance behavior: (i) relying on AI recommendations in too few cases (i.e., under-reliance, see [10,11], e.g., by underestimating AI performance), (ii) relying on AI recommendations in too many cases (i.e., over-reliance, see [1,12,13], e.g., by overestimating AI performance), and (iii) relying appropriately on AI recommendations (i.e., adhering to AI recommendations when correct and overriding when wrong, see [5,9,14]). Thus far, research has identified many scenarios in which underreliance or over-reliance results in reduced decision-making performance (e.g., [12,15]).…”

Section: Introductionmentioning

confidence: 99%

“…Accuracy of AI-assisted decision-making (i.e., the number of correct decisions given the overall number of decisions) represents a key metric that may indicate the utility of an AI-assisted decision-making system-apart from other metrics such as fairness [5]. The accuracy metric is hence frequently used for measuring the performance of AI-assisted decision-making systems [3] and evaluating the effectiveness of interventions (e.g., explanations) for decision support [3,16,17,18].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

Schoeffer

Jakubik

Voessing

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

In AI-assisted decision-making, a central promise of putting a human in the loop is that they should be able to complement the AI system by adhering to its correct and overriding its mistaken recommendations. In practice, however, we often see that humans tend to over- or under-rely on AI recommendations, meaning that they either adhere to wrong or override correct recommendations. Such reliance behavior is detrimental to decision-making accuracy. In this work, we articulate and analyze the interdependence between reliance behavior and accuracy in AI-assisted decision-making, which has been largely neglected in prior work. We also propose a visual framework to make this interdependence more tangible. This framework helps us interpret and compare empirical findings, as well as obtain a nuanced understanding of the effects of interventions (e.g., explanations) in AI-assisted decision-making. Finally, we infer several interesting properties from the framework: (i) when humans under-rely on AI recommendations, there may be no possibility for them to complement the AI in terms of decision-making accuracy; (ii) when humans cannot discern correct and wrong AI recommendations, no such improvement can be expected either; (iii) interventions may lead to an increase in decision-making accuracy that is solely driven by an increase in humans’ adherence to AI recommendations, without any ability to discern correct and wrong. Our work emphasizes the importance of measuring and reporting both effects on accuracy and reliance behavior when empirically assessing interventions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

Schoeffer

Jakubik

Voessing

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…Empirical evidence supports that explanations aid human cognition in identifying system failures, promoting what is termed calibrated trust [57]. However, a counter-narrative warns against excessive reliance on machine explanations, as users may endorse AI results despite errors [4,61].…”

Section: Introductionmentioning

confidence: 99%

Explaining the Unexplainable: The Impact of Misleading Explanations on Trust in Unreliable Predictions for Hardly Assessable Tasks

Sadeghi,

Pöttgen,

Ebel

et al. 2024

Preprint

View full text Add to dashboard Cite

To increase trust in systems, engineers strive to create explanations that are as accurate as possible. However, if the system's accuracy is compromised, providing explanations for its incorrect behavior may inadvertently lead to misleading explanations. This concern is particularly pertinent when the correctness of the system is difficult for users to judge. In an online survey experiment with 162 participants, we analyze the impact of misleading explanations on users' perceived and demonstrated trust in a system that performs a hardly assessable task in an unreliable manner. Participants who used a system that provided potentially misleading explanations rated their trust significantly higher than participants who saw the system's prediction alone. They also aligned their initial prediction with the system's prediction significantly more often. Our findings underscore the importance of exercising caution when generating explanations, especially in tasks that are inherently difficult to evaluate. The paper and supplementary materials are available at https://doi.org/10.17605/osf.io/azu72.

show abstract

A signal detection perspective on error and unfairness detection as a critical aspect of human oversight of AI-based systems

Langer¹,

Baum²,

Schlicker³

2023

Preprint

View full text Add to dashboard Cite

Legislation and ethical guidelines around the globe call for effective human oversight of the use of AI-based systems in high-risk contexts – that is oversight that reduces the risks associated with the use of AI-based systems. Such risks may relate to the imperfect accuracy of systems (e.g., incorrect classifications) or to ethical concerns (e.g., unfairness of outputs). Given the significant role that human oversight is expected to play in the operation of AI-based systems, it is crucial to gain a better understanding of the conditions for effective oversight. In this paper, we build on the literature on the management of imperfect automation to show that the reliable detection of errors or other deviant behavior is crucial for the effective management of these errors and thus crucial for effective oversight. We then propose that Signal Detection Theory (SDT) offers a promising framework for better understanding what affects people’s sensitivity (i.e., how well they are able to detect errors) and response bias (i.e., the tendency to report errors given a perceived evidence of an error) in detecting errors. To demonstrate the broad applicability of an SDT perspective to the study of error detection when overseeing AI-based systems, we then explicate the specifics for the case of unfairness detection. Additionally, we propose factors (task-, system-, and person-related factors) that may affect the sensitivity and response bias of humans tasked with detecting unfairness associated with the use of AI-based systems. Finally, we discuss implications and future research directions for an SDT perspective on error detection.

show abstract

On Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making

Cited by 4 publications

References 0 publications

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

Explaining the Unexplainable: The Impact of Misleading Explanations on Trust in Unreliable Predictions for Hardly Assessable Tasks

A signal detection perspective on error and unfairness detection as a critical aspect of human oversight of AI-based systems

Contact Info

Product

Resources

About