To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis

Abstract

Artificial intelligence (AI) technologies promise to transform how professionals conduct knowledge work by augmenting their capabilities for making professional judgments. We know little, however, about how human-AI augmentation takes place in practice. Yet, gaining this understanding is particularly important when professionals use AI tools to form judgments on critical decisions. We conducted an in-depth field study in a major U.S. hospital where AI tools were used in three departments by diagnostic radiologists making breast cancer, lung cancer, and bone age determinations. The study illustrates the hindering effects of opacity that professionals experienced when using AI tools and explores how these professionals grappled with it in practice. In all three departments, this opacity resulted in professionals experiencing increased uncertainty because AI tool results often diverged from their initial judgment without providing underlying reasoning. Only in one department (of the three) did professionals consistently incorporate AI results into their final judgments, achieving what we call engaged augmentation. These professionals invested in AI interrogation practices—practices enacted by human experts to relate their own knowledge claims to AI knowledge claims. Professionals in the other two departments did not enact such practices and did not incorporate AI inputs into their final decisions, which we call unengaged “augmentation.” Our study unpacks the challenges involved in augmenting professional judgment with powerful, yet opaque, technologies and contributes to literature on AI adoption in knowledge work.

Why Providing Humans with Interpretable Algorithms May, Counterintuitively, Lead to Lower Decision-making Performance

“…Second, algorithms differ in their interpretability. Compared to ruled-based algorithms, which are based on hierarchies of rules and control flows (Lebovitz et al, 2022), ML algorithms are less intelligible to users, who have difficulties understanding the quality of the information received. The reduction of algorithmic interpretability correlates to the increasing mathematical knowledge required to understand the model and the information processing required for humans to replicate algorithmic paths and rules (Burrell, 2016).…”

Pragmatic AI-augmentation in mental healthcare: Key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians

“…Clinical leaders could highlight the essential role played by humans at each phase of the development and implementation of an AI technology ( 30 ). For example, leaders could note that radiologists may relinquish the preliminary scanning of medical images to the AI technology in order to focus on the more complicated cases, or those that the algorithm finds ambiguous ( 31 ); technology developers can include feedback during screening that recommends reaching out for in-person evaluation to verify the clinical recommendation ( 32 ).…”

“…First, regarding clinician daily work practices, ML model outputs are often complex and uninterpretable to humans—the so called ‘black box' problem. The precise reason for an ML model's recommendation often cannot be easily pinpointed, even by data scientists ( 31 ). In addition, technology-based approaches are pattern-based, while traditional research methods are hypothesis-driven, and most clinicians do not have the training required to engage in the computational thinking associated with these interpretations ( 17 , 37 ).…”

Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making

“…The output it generates, however, is often opaque and difficult to interpret. Machine learning as a black box creates "epistemic uncertainty and opacity" [Lebovitz et al 2022]. With the proliferation of machine learning models in all spheres of life there is a perceived need for mechanisms that can help users and other stakeholders to better understand algorithmic predictions.…”