Human–AI Interactions in Public Sector Decision Making: “Automation Bias” and “Selective Adherence” to Algorithmic Advice

Abstract

Artificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases of human decision-makers. At the same time, they may introduce new biases in the human-algorithm interaction. Drawing on psychology and public administration literatures, we investigate two key biases: overreliance on algorithmic advice even in the face of ‘warning signals’ from other sources (automation bias), and selective adoption of algorithmic advice when this corresponds to stereotypes (selective adherence). We assess these via three experimental studies conducted in the Netherlands: In study 1 (N=605), we test automation bias by exploring participants’ adherence to an algorithmic prediction compared to an equivalent human-expert prediction. We do not find evidence for automation bias. In study 2 (N=904), we replicate these findings, and also test selective adherence. We find a stronger propensity for adherence when the advice is aligned with group stereotypes, with no significant differences between algorithmic and human-expert advice. Studies 1 and 2 were conducted among citizens in a context where citizens can act as decision-makers. In study 3 (N=1,345), we replicate our design with a sample of civil servants. This study was conducted shortly after a major scandal involving public authorities’ reliance on an algorithm with discriminatory outcomes (the “childcare benefits scandal”). The scandal is itself illustrative of our theory and patterns diagnosed empirically in our experiment, yet in our study 3, while supporting our prior findings as to automation bias, we do not find patterns of selective adherence. We suggest this is driven by bureaucrats’ enhanced awareness of discrimination and algorithmic biases in the aftermath of the scandal. We discuss the implications of our findings for public sector decision-making in the age of automation. Overall, our study speaks to potential negative effects of automation of the administrative state for already vulnerable and disadvantaged citizens.

AI algorithmic oversight: new frontiers in regulation

“…Namely, decision-makers adhere to the algorithmic advice (rather than resist it) precisely when predictions are aligned with prevalent societal stereotypes. These experimental findings are consistent with patterns observed in real-life settings for example, in the Dutch childcare benefits scandal, mentioned above, ethnic minority citizens (of Moroccan, Turkish and Dutch Antilles origin) were disproportionately impacted (Financial Times 2021): erroneous algorithmic predictions aligned with prevalent stereotypes, with bureaucratic decision-makers unlikely to override such predictions (see further Alon-Barkat & Busuioc 2022).…”

“…The influence that AI algorithms have on human decision-makers is poorly understood, and various studies have raised the prospect that human overseers (decisional mediators) could be prone to important cognitive biases in this respect such as "automation bias", as discussed above. Recent research raises the prospect of additional cognitive biases that can arise in human processing of AI algorithmic outputs: for instance, that human decision-makers are inclined to give more weight and defer to algorithmic recommendations that align with decision-makers' worldviews, with what they already believe to be true that is, when predictions conform with pre-existing beliefs and stereotypes (Alon-Barkat & Busuioc 2022). This could invertedly lead them to make decisions that are harmful and can compound (rather than mitigate) bias.…”

“…This could invertedly lead them to make decisions that are harmful and can compound (rather than mitigate) bias. In a recent empirical study on the use of AI algorithms as decisional aides, Alon- Barkat & Busuioc (2022), diagnose a tendency among human decision-makers to follow AI algorithmic advice selectively, in a biased manner, when this advice matches pre-existing stereotypical beliefs of decision subjects. Namely, decision-makers adhere to the algorithmic advice (rather than resist it) precisely when predictions are aligned with prevalent societal stereotypes.…”

Can AI Reduce Motivated Reasoning in News Consumption? Investigating the Role of Attitudes Towards AI and Prior-Opinion in Shaping Trust Perceptions of News

“…In other words, the studies do not illuminate whether the effects of congruency are unique to AI or whether we can expect a preference for any (AI and human) actor who shows congruency. Overcoming this limitation, [28] found that congruency increased adherence to AI-assisted recommendations but to a similar degree as recommendations by a co-worker. Hence, [28] could show that individuals did not differentiate between AI and a co-worker in terms of motivated reasoning.…”

“…Overcoming this limitation, [28] found that congruency increased adherence to AI-assisted recommendations but to a similar degree as recommendations by a co-worker. Hence, [28] could show that individuals did not differentiate between AI and a co-worker in terms of motivated reasoning. Receiving information from both, AI and the co-worker, resulted in the same pattern of motivated reasoning.…”