Usability and Clinician Acceptance of a Deep Learning-Based Clinical Decision Support Tool for Predicting Glaucomatous Visual Field Progression

Abstract

Précis: We updated a clinical decision support tool integrating predicted visual field (VF) metrics from an artificial intelligence model and assessed clinician perceptions of the predicted VF metric in this usability study. Purpose: To evaluate clinician perceptions of a prototyped clinical decision support (CDS) tool that integrates visual field (VF) metric predictions from artificial intelligence (AI) models. Methods: Ten ophthalmologists and optometrists from the University of California San Diego participated in 6 cases from 6 patients, consisting of 11 eyes, uploaded to a CDS tool (“GLANCE”, designed to help clinicians “at a glance”). For each case, clinicians answered questions about management recommendations and attitudes towards GLANCE, particularly regarding the utility and trustworthiness of the AI-predicted VF metrics and willingness to decrease VF testing frequency. Main Outcome(s) and Measure(s): Mean counts of management recommendations and mean Likert scale scores were calculated to assess overall management trends and attitudes towards the CDS tool for each case. In addition, system usability scale scores were calculated. Results: The mean Likert scores for trust in and utility of the predicted VF metric and clinician willingness to decrease VF testing frequency were 3.27, 3.42, and 2.64, respectively (1=strongly disagree, 5=strongly agree). When stratified by glaucoma severity, all mean Likert scores decreased as severity increased. The system usability scale score across all responders was 66.1±16.0 (43rd percentile). Conclusions: A CDS tool can be designed to present AI model outputs in a useful, trustworthy manner that clinicians are generally willing to integrate into their clinical decision-making. Future work is needed to understand how to best develop explainable and trustworthy CDS tools integrating AI before clinical deployment.

Review of Visualization Approaches in Deep Learning Models of Glaucoma

“…They found that while their interface was revealed to show mediocre usability (SUS score in the 43rd percentile, mean ± SD SUS score = 66.1±16.0), earlier work has shown that clinicians commonly have unfavorable perceptions of EHR usability when estimated using SUS scores (mean scores < 10th percentile). This highlights the challenges of developing usable CDS instruments in the EHR and the demand for continuous work in this area 71 …”

“…70 These instruments are a reliable way to evaluate user satisfaction with DL models and to assess the cognitive and time burden a model imposes. 70 Chen et al 71 recently applied one of these tools, the SUS score, to calculate a commonly used and validated scoring system that ranges from 0 to 100 for evaluating the user-friendliness of the GLANCE interface. They found that while their interface was revealed to show mediocre usability (SUS score in the 43rd percentile, mean ± SD SUS score = 66.1 ± 16.0), earlier work has shown that clini cians commonly have unfavorable perceptions of EHR usability when estimated using SUS scores (mean scores < 10th percentile).…”

“…This highlights the challenges of developing usable CDS instruments in the EHR and the demand for continuous work in this area. 71 Integration into clinical workflows is also essential to ensure that AI models are seamlessly incorporated into existing clinical information systems (such as existing EHR and picture archiving and communications systems), rather than residing in external systems that take additional time and effort for clinicians to access. This requires the adoption and implementation of data standards to enable clinical operability.…”