Algorithmic Harms in Child Welfare: Uncertainties in Practice, Organization, and Street-level Decision-making

Abstract

Algorithms in public services such as child welfare, criminal justice, and education are increasingly being used to make high-stakes decisions about human lives. Drawing upon findings from a two-year ethnography conducted at a child welfare agency, we highlight how algorithmic systems are embedded within a complex decision-making ecosystem at critical points of the child welfare process. Caseworkers interact with algorithms in their daily lives where they must collect information about families and feed it to algorithms to make critical decisions. We show how the interplay between systemic mechanics and algorithmic decision-making can adversely impact the fairness of the decision-making process itself. We show how functionality issues in algorithmic systems can lead to process-oriented harms where they adversely affect the nature of professional practice, and administration at the agency, and lead to inconsistent and unreliable decisions at the street level. In addition, caseworkers are compelled to undertake additional labor in the form of repair work to restore disrupted administrative processes and decision-making, all while facing organizational pressures and time and resource constraints. Finally, we share the case study of a simple algorithmic tool that centers caseworkers’ decision-making within a trauma-informed framework and leads to better outcomes, however, required a significant amount of investments on the agency’s part in creating the ecosystem for its proper use.

"This is not a data problem": Algorithms and Power in Public Higher Education in Canada

“…We used open-coding to identify the highstakes outcomes for which student data-driven and algorithmic decision-making are leveraged and how data and algorithms are currently used within the college's work practices. We then adopted the ADMAPS framework to reveal the complex inter-dependencies between ADMAPS dimensions [53], coding out data to human discretion, bureaucratic processes and algorithmic decision-making [51]. Finally, we analyzed the resulting patterns to identify emergent themes to construct a holistic view of the role of data and algorithms in the institution.…”

Are We Asking the Right Questions?: Designing for Community Stakeholders’ Interactions with AI in Policing

“…The deployment and use of algorithms in these sectors have far-reaching consequences, often impacting lives and, in some cases, resulting in life-altering or life-and-death outcomes. As a result, SIGCHI researchers have become increasingly interested in understanding how human-AI decision-making unfolds in these domains [75,111], the perspective of frontline workers whose labor is impacted by these technologies [3,29], the impact of organizational and policy-related factors [113], as well as unpacking implications for the worker-centered design of ADS in these contexts [76,115].…”

“…Similarly, Guerdan et al [55] reviewed several case studies of AI systems in the public sector and developed a causal framework to identify sources of target variable bias and their impact on human-AI decision-making. In addition, researchers have also explored opportunities to support collaboration in human-AI decision-making [21,50,82] by developing explanations [25], training protocols [74], and prescribing away from predicting outcomes [113].…”

“…However, growing evidence suggests that these systems exacerbate human biases embedded in the historical data [20,87,134] as well as introduce new biases as a result of the inconsistency between the workers' decision-making latitude versus the empirical decisions recommended by the ADS [29,102,111]. These issues have underscored the calls for developing human-centered systems [4,67,110] that situate empirical predictions in the context of work and complement workers' expertise [74,113], i.e. -allow stakeholders such as law enforcement and crime analysts to derive meaningful information that aligns with their work and informs decision-making on interventions and resource allocation.…”

“…-community members. Here, SIGCHI researchers have begun collaborating directly with community members affected by ADS [81,93,125,141], in addition to civil servants whose work is impacted by such technology [3,74,113]. Engaging with a diverse range of community stakeholders enables us to gain a deeper understanding of what it means for ADS used in government agencies to operate in the public interest and further helps us identify ways to enhance the practices of civil servants who are entrusted with serving the community [125,136].…”