Collaborative Human-AI Sensemaking for Intelligence Analysis

Journal of Cognitive Engineering and Decision Making

2022

Self Cite

Artificial Intelligence (AI) is often viewed as the means by which the intelligence community will cope with increasing amounts of data. There are challenges in adoption, however, as outputs of such systems may be difficult to trust, for a variety of factors. We conducted a naturalistic study using the Critical Incident Technique (CIT) to identify which factors were present in incidents where trust in an AI technology used in intelligence work (i.e., the collection, processing, analysis, and dissemination of intelligence) was gained or lost. We found that explainability and performance of the AI were the most prominent factors in responses; however, several other factors affected the development of trust. Further, most incidents involved two or more trust factors, demonstrating that trust is a multifaceted phenomenon. We also conducted a broader thematic analysis to identify other trends in the data. We found that trust in AI is often affected by the interaction of other people with the AI (i.e., people who develop it or use its outputs), and that involving end users in the development of the AI also affects trust. We provide an overview of key findings, practical implications for design, and possible future areas for research.

Section: Trust In Aimentioning

confidence: 99%

mentioning

confidence: 99%

A Naturalistic Investigation of Trust, AI, and Intelligence Work

Journal of Cognitive Engineering and Decision Making

2022

Self Cite

“…Despite the promise of AI and its numerous applications across the domain of intelligence analysis, Dorton and Harper (2021) argue that even the most powerful AI capabilities are meaningless if analysts and their downstream customers (e.g. leaders in military decision making or statecraft) do not trust the AI or its outputs.…”

Section: Introduction Ai In Intelligence Analysismentioning

confidence: 99%

Self-Repairing and/or Buoyant Trust in Artificial Intelligence

Proceedings of the Human Factors and Ergonomics Society Annual

2022

Artificial Intelligence (AI) is often viewed as the means by which the intelligence community will cope with the increasing amount of information available to them. Trust is a complex, dynamic phenomenon, which drives adoption (or disuse) of technology. We conducted a naturalistic study with intelligence professionals (planners, collectors, analysts, etc.) to understand trust dynamics with AI systems. We found that on a long-enough time scale, trust in AI self-repaired after incidents where trust was lost, usually based merely on the assumption that AI had improved since participants last interacted with it. Similarly, we found that trust in AI increased over time after incidents where trust was gained in the AI. We termed this general trend “buoyant trust in AI,” where trust in AI tends to increase over time, regardless of previous interactions with the system. Key findings are discussed, along with possible directions for future research.

“…Trust in AI more specifically has been viewed through the lens of trust being a function of explainability, performance (e.g. minimal false positives), and reliability (Hoffman et al, 2018;Lyons et al 2016); however, recent research has shown that there are numerous factors affecting trust in AI (Dorton, & Harper, 2021;Dorton & Harper, 2022).…”

Section: Trust (In Artificial Intelligence)mentioning

confidence: 99%

“…) are proverbially drowning in data. Artificial Intelligence (AI) is proposed as a means by which we can cope with the deluge of data (Symon & Tarapore, 2015); however, there are several factors that affect trust in AI, and it has been argued that these powerful technologies are moot if they are not trustable in such high stakes contexts (Dorton & Harper, 2021). Further, humans often adapt to new technology in complex work systems (Patterson, et al, 2004).…”

Section: Introductionmentioning

confidence: 99%

Adaptations to Trust Incidents with Artificial Intelligence

Proceedings of the Human Factors and Ergonomics Society Annual

Neville

2022

Artificial Intelligence (AI) is becoming ubiquitous in national security work (intelligence, defense, etc.); however, introducing AI into work systems is fraught with challenges. Trust is gained and lost through experiences, and there are many factors that affect trust in AI. Similarly, users adapt their workflows based on trust in these systems. We used a naturalistic approach to understand how intelligence professionals adapted their work practices after gaining or losing trust in AI. We found a variety of adaptations, which were characterized as either being task-based or frequency-based; where users added or removed tasks from their workflow or where they changed the frequency in which they used the AI in their workflow, respectively. We provide specific examples and quotes from participants along with findings, and discuss potential methodological implications for studying and designing AI-driven work systems.