Initial Responses to False Positives in AI-Supported Continuous Interactions: A Colonoscopy Case Study

Abstract

The use of artificial intelligence (AI) in clinical support systems is increasing. In this article, we focus on AI support for continuous interaction scenarios. A thorough understanding of end-user behaviour during these continuous human-AI interactions, in which user input is sustained over time and during which AI suggestions can appear at any time, is still missing. We present a controlled lab study involving 21 endoscopists and an AI colonoscopy support system. Using a custom-developed application and an off-the-shelf videogame controller, we record participants’ navigation behaviour and clinical assessment across 14 endoscopic videos. Each video is manually annotated to mimic an AI recommendation, being either true positive or false positive in nature. We find that time between AI recommendation and clinical assessment is significantly longer for incorrect assessments. Further, the type of medical content displayed significantly affects decision time. Finally, we discover that the participant’s clinical role plays a large part in the perception of clinical AI support systems. Our study presents a realistic assessment of the effects of imperfect and continuous AI support in a clinical scenario.

Measurements, Algorithms, and Presentations of Reality: Framing Interactions with AI-Enabled Decision Support

“…While existing recommendations on Human-AI interaction have long stressed the need for AI suggestions to be accompanied by explanations (see e.g., Amershi et al [1]), such explanations can also be experienced as interrupting or even harmful when the user is currently mentally and physically occupied, for example, during surgery [103].…”

“…As well as referencing existing literature (above), we present two case studies, both of which involved exploratory AI systems aimed at experts in the healthcare sector. The first case study, which focuses on colonoscopy practice, is part of a project that aims to support endoscopists in identifying polyps in the colon during an inspection [99,103]. These polyps are often challenging to identify, with up to 27% of polyps left unidentified [105,125].…”

“…In a second study, we assessed the impact of AI recommendations on clinicians during medical procedures [103]. Here, we presented endoscopists with pre-recorded endoscopic videos, which we overlaid with manually annotated recommendations to mimic the behaviour of an AI system.…”

“…For example, false positives result in increased clinical effort-while the closer inspection of an area can be considered a positive side-effect, it also leads to higher patient discomfort and healthcare costs. [103]. Our work focused specifically on the impact of false positives (i.e., an AI suggestion in case of non-polyps) [103].…”

“…[103]. Our work focused specifically on the impact of false positives (i.e., an AI suggestion in case of non-polyps) [103]. Clinical outcomes could be severely impacted if clinicians fail to recognise false negatives (i.e., the AI support failing to pick up on polyps that are present) due to increased automation complacency.…”