Adaptations to Trust Incidents with Artificial Intelligence

Abstract

Artificial Intelligence (AI) is becoming ubiquitous in national security work (intelligence, defense, etc.); however, introducing AI into work systems is fraught with challenges. Trust is gained and lost through experiences, and there are many factors that affect trust in AI. Similarly, users adapt their workflows based on trust in these systems. We used a naturalistic approach to understand how intelligence professionals adapted their work practices after gaining or losing trust in AI. We found a variety of adaptations, which were characterized as either being task-based or frequency-based; where users added or removed tasks from their workflow or where they changed the frequency in which they used the AI in their workflow, respectively. We provide specific examples and quotes from participants along with findings, and discuss potential methodological implications for studying and designing AI-driven work systems.

Foresight for ethical AI

“…This value extends beyond developing ethical AI. NDM approaches have been used to glean insights on developing trustworthy AI (e.g., Dorton andDorton, 2022) and resilient AI technologies (e.g., Neville et al, 2022). In summary, we have attempted to craft our argument while providing different messages to two different audiences.…”

“…Emergent behaviors or properties can be defined as those arising from interactions among components in a sociotechnical system, that are otherwise not observable when investigating any one component (Carayon et al, 2015 ). Such behaviors exist at the individual level, where introducing AI into a workflow can increase human complacency (Ezer et al, 2019 ), or drive workflow adaptations based on changes in trust with the AI over time (Dorton et al, 2022 ). There is also the issue of complexity at the higher, sociotechnical system level.…”

Exploring Trust With the AI Incident Database

“…The Naturalistic Decision Making (NDM) tradition focuses on developing understanding of human cognition situated in the context of work (vs in a controlled lab setting) and enables the engineering of systems to support cognitive processes (Klein et al, 1993). Naturalistic inquiry has been effectively applied to glean insights on how trust is formed in AI in high consequence work (Rotner et al, 2021; Dorton & Harper, 2022a; Dorton, 2022), work adaptations to changes in trust (Dorton et al, 2022), and log-term trust dynamics with AI (Dorton & Harper, 2022).…”

RAD-XP: Tabletop Exercises for Eliciting Resilience Requirements for Sociotechnical Systems

“…We often focus on those goals and assume the technology will be adopted into use; however, any technology injected into the complex, time-honed dynamics of an established sociotechnical system is unlikely to be prepared for participating effectively in those dynamics. As such, it is unfortunately common that the introduction of new technologies to an existing sociotechnical system can have adverse unintended effects [6][7][8].…”