Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support

Abstract

AI-based decision support tools (ADS) are increasingly used to augment human decision-making in high-stakes, social contexts. As public sector agencies begin to adopt ADS, it is critical that we understand workers' experiences with these systems in practice. In this paper, we present findings from a series of interviews and contextual inquiries at a child welfare agency, to understand how they currently make AI-assisted child maltreatment screening decisions. Overall, we observe how workers' reliance upon the ADS is guided by (1) their knowledge of rich, contextual information beyond what the AI model captures, (2) their beliefs about the ADS's capabilities and limitations relative to their own, (3) organizational pressures and incentives around the use of the ADS, and (4) awareness of misalignments between algorithmic predictions and their own decision-making objectives. Drawing upon these findings, we discuss design implications towards supporting more effective human-AI decision-making.

Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders

“…Against predictive algorithms in CPS. Our participants gave more novel suggestions and critical feedback than in prior participatory work with impacted communities and workers in CPS [20,26,27,67,68,115]. For example, Brown et al [20] suggest their participants' "general distrust in the existing system" (which they somewhat vaguely describe as "system-level concerns") led to "low comfort in algorithmic decision-making," and suggested these problems could be improved through "greater transparency and improved communication strategies. "…”

“…While more common across HCI and CSCW, less work in participatory ML empowers stakeholders to decide on the "scope and purpose for AI, including whether it should be built or not" [36]. Specifically around the design of algorithms in child welfare, 2 prior participatory work has either collaborated with government agencies or solely engaged with government workers in their studies [20,26,67,68,115]. 3 Most similar to our work, Brown et al [20] partnered with a CPS agency to aid the development of a PRM by conducting participatory design workshops where they asked workers and community stakeholders about scenarios related to specific design choices.…”

“…then asked the lead author to consider whether they were doing this work to publish a paper and further their academic career or whether it was work which could actually benefit families harmed by CPS. 12 Given that most prior work on CPS in ML and HCI has been conducted to help develop algorithms to assess families or in partnership with CPS agencies [20,26,67,114,115], these suspicions seem justified. Instead, participants suggested specific ways researchers could better work in solidarity with communities.…”

“…A machine learning (ML) algorithm then uses this data to train a model (the PRM). Finally, this PRM is applied to new case data and the PRM's assessment -interpreted as the likelihood of some proxy for abuse or neglect-is shown to CPS workers, who use it when making decisions [67,113]. Although no PRM is currently used to fully automate decisions, some suggest this is possible [26,35,45,84].…”

“…Although no PRM is currently used to fully automate decisions, some suggest this is possible [26,35,45,84]. Others note that automation is a spectrum: CPS agencies can pressure workers to conform to PRMs' recommendations in some cases more than others [26,67].…”