Surveillance, Discretion and Governance in Automated Welfare

Abstract

Several scholarly studies and journalistic investigations have found that automated decision-making in welfare systems burdens claimants by forecasting their behaviour, targeting them for sanctions and surveillance and punishing them without revealing the underlying mechanisms driving such decisions. This article develops an analytical framework combining three areas of concern regarding automation: how it might introduce surveillance and social sorting, how it can entail the loss of human discretion, and how it requires new systems of governance and due process. This framework steers investigations into whether and how automated decision-making welfare systems introduce new harms and burdens for claimants. A case study on automation processes in Germany’s unemployment benefit service’s IT system ALLEGRO applies this approach and finds that this system does allow for broad human discretion and avoids some forms of surveillance, such as risk-assessments from historic data, though it nevertheless increases surveillance of claimants through sharing data with external agencies. The developed framework also suggests that concerns raised in one area – whether loss of human discretion, surveillance, or lack of due process – can be mitigated by attending to the other two areas and urges researchers and policy-makers to attend to the mitigating or reinforcing factors of each concern.

Emotions and Dynamic Assemblages: A Study of Automated Social Security Using Qualitative Longitudinal Research

“…In addition to concerns around surveillance, agency, discretion, and governance [77], there are epistemological challenges that scholars often face when studying ADMS [31] [37] [28]. In the private sector, recommendation engines on social media and search engines are often tightly guarded by trade secrets.…”