Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods; User studies.

People predict that AI agents will be as benevolent (cooperative) as humans People cooperate less with benevolent AI agents than with benevolent humans Reduced cooperation only occurs if it serves people's selfish interests People feel guilty when they exploit humans but not when they exploit AI agents

No abstract