Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

Druce, Jeff; Niehaus, James; Moody, Vanessa; Jensen, David G.; Littman, Michael L.

doi:10.48550/arxiv.2106.05506

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, it has been pointed out that explainability alone does not fully meet expectations and does not guarantee the achievement of the objectives for which it was theorized [20]. This is even clearer if we analyze the legal principles that algorithmic intelligibility would be required to pursue.…”

Section: Explainable Aimentioning

confidence: 93%

Risk and Exposure of XAI in Persuasion and Argumentation: The case of Manipulation

Carli

Najjar²,

Calvaresi³

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In the last decades, Artificial intelligence (AI) systems have been increasingly adopted in assistive (possibly collaborative) decisionmaking tools. In particular, AI-based persuasive technologies are designed to steer/influence users' behaviour, habits, and choices to facilitate the achievement of their own -predetermined -goals. Nowadays, the inputs received by the assistive systems leverage heavily AI data-driven approaches. Thus, it is imperative to have transparent and understandable (to the user) both the process leading to the recommendations and the recommendations. The Explainable AI (XAI) community has progressively contributed to "opening the black box", ensuring the interaction's effectiveness, and pursuing the safety of the individuals involved. However, principles and methods ensuring the efficacy and information retain on the human have not been introduced yet. The risk is to underestimate the context dependency and subjectivity of the explanations' understanding, interpretation, and relevance. Moreover, even a plausible (and possibly expected) explanation can lead to an imprecise or incorrect outcome or its understanding. This can lead to unbalanced and unfair circumstances, such as giving a financial advantage to the system owner/provider and the detriment of the user. This paper highlights that the sole explanations -especially in the context of persuasive technologies -are not self-sufficient to protect users' psychological and physical integrity. Conversely, explanations could be misused, becoming themselves a tool of manipulation. Therefore, we suggest characteristics safeguarding the explanation from being manipulative and legal principles to be used as criteria for evaluating the operation of XAI systems, both from an ex-ante and ex-post perspective.

show abstract

Section: Explainable Aimentioning

confidence: 93%

Risk and Exposure of XAI in Persuasion and Argumentation: The case of Manipulation

Carli

Najjar²,

Calvaresi³

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Explanations could help individuals to recognize errors in their mental model, leading to a better fit between experienced traceability and performance. However, they could also erroneously increase the confidence in an incorrect mental model and thus worsen the calibration [34], which results in wrong expectations about system behavior and potentially confuses users, ultimately leading to a reduction of trust [109]. Explanations can have an ambiguous effect on the calibration between experienced traceability of a system and the user's ability to correctly predict information processing.…”

mentioning

confidence: 99%

How do Users Experience Traceability of AI Systems? Examining Subjective Information Processing Awareness in Automated Insulin Delivery (AID) Systems

Schrills¹,

Franke²

2022

Preprint

View full text Add to dashboard Cite

When interacting with artificial intelligence (AI) in the medical domain, users frequently face automated information processing, which can remain opaque to them. For example, users with diabetes may interact daily with automated insulin delivery (AID). However, effective AID therapy requires traceability of automated decisions for diverse users. Grounded in research on human-automation interaction, we study Subjective Information Processing Awareness (SIPA) as a key construct to research users' experience of explainable AI. The objective of the present research was to examine how users experience differing levels of traceability of an AI algorithm. We developed a basic AID simulation to create realistic scenarios for an experiment with N = 80, where we examined the effect of three levels of information disclosure on SIPA and performance. Attributes serving as the basis for insulin needs calculation were shown to users, who predicted the AID system's calculation after over 60 observations. Results showed a difference in SIPA after repeated observations, associated with a general decline of SIPA ratings over time. Supporting scale validity, SIPA was strongly correlated with trust and satisfaction with explanations. The present research indicates that the effect of different levels of information disclosure may need several repetitions before it manifests. Additionally, high levels of information disclosure may lead to a miscalibration between SIPA and performance in predicting the system's results. The results indicate that for a responsible design of XAI, system designers could utilize prediction tasks in order to calibrate experienced traceability.

show abstract

Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Vasconcelos

Jörke

Grunde-McLaughlin

et al. 2023

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Prior work has identified a resilient phenomenon that threatens the performance of human-AI decision-making teams: overreliance, when people agree with an AI, even when it is incorrect. Surprisingly, overreliance does not reduce when the AI produces explanations for its predictions, compared to only providing predictions. Some have argued that overreliance results from cognitive biases or uncalibrated trust, attributing overreliance to an inevitability of human cognition. By contrast, our paper argues that people strategically choose whether or not to engage with an AI explanation, demonstrating empirically that there are scenarios where AI explanations reduce overreliance. To achieve this, we formalize this strategic choice in a cost-benefit framework, where the costs and benefits of engaging with the task are weighed against the costs and benefits of relying on the AI. We manipulate the costs and benefits in a maze task, where participants collaborate with a simulated AI to find the exit of a maze. Through 5 studies (N = 731), we find that costs such as task difficulty (Study 1), explanation difficulty (Study 2, 3), and benefits such as monetary compensation (Study 4) affect overreliance. Finally, Study 5 adapts the Cognitive Effort Discounting paradigm to quantify the utility of different explanations, providing further support for our framework. Our results suggest that some of the null effects found in literature could be due in part to the explanation not sufficiently reducing the costs of verifying the AI's prediction.

show abstract

Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

Cited by 3 publications

References 5 publications

Risk and Exposure of XAI in Persuasion and Argumentation: The case of Manipulation

Risk and Exposure of XAI in Persuasion and Argumentation: The case of Manipulation

How do Users Experience Traceability of AI Systems? Examining Subjective Information Processing Awareness in Automated Insulin Delivery (AID) Systems

Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Contact Info

Product

Resources

About