Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation

Abstract

Background Poor usability is a primary cause of unintended consequences related to the use of electronic health record (EHR) systems, which negatively impacts patient safety. Due to the cost and time needed to carry out iterative evaluations, many EHR components, such as clinical decision support systems (CDSSs), have not undergone rigorous usability testing prior to their deployment in clinical practice. Usability testing in the predeployment phase is crucial to eliminating usability issues and preventing costly fixes that will be needed if these issues are found after the system’s implementation. Objective This study presents an example application of a systematic evaluation method that uses clinician experts with human-computer interaction (HCI) expertise to evaluate the usability of an electronic clinical decision support (CDS) intervention prior to its deployment in a randomized controlled trial. Methods We invited 6 HCI experts to participate in a heuristic evaluation of our CDS intervention. Each expert was asked to independently explore the intervention at least twice. After completing the assigned tasks using patient scenarios, each expert completed a heuristic evaluation checklist developed by Bright et al based on Nielsen’s 10 heuristics. The experts also rated the overall severity of each identified heuristic violation on a scale of 0 to 4, where 0 indicates no problems and 4 indicates a usability catastrophe. Data from the experts’ coded comments were synthesized, and the severity of each identified usability heuristic was analyzed. Results The 6 HCI experts included professionals from the fields of nursing (n=4), pharmaceutical science (n=1), and systems engineering (n=1). The mean overall severity scores of the identified heuristic violations ranged from 0.66 (flexibility and efficiency of use) to 2.00 (user control and freedom and error prevention), in which scores closer to 0 indicate a more usable system. The heuristic principle user control and freedom was identified as the most in need of refinement and, particularly by nonnursing HCI experts, considered as having major usability problems. In response to the heuristic match between system and the real world, the experts pointed to the reversed direction of our system’s pain scale scores (1=severe pain) compared to those commonly used in clinical practice (typically 1=mild pain); although this was identified as a minor usability problem, its refinement was repeatedly emphasized by nursing HCI experts. Conclusions Our heuristic evaluation process is simple and systematic and can be used at multiple stages of system development to reduce the time and cost needed to establish the usability of a system before its widespread implementation. Furthermore, heuristic evaluations can help organizations develop transparent reporting protocols for usability, as required by Title IV of the 21st Century Cures Act. Testing of EHRs and CDSSs by clinicians with HCI expertise in heuristic evaluation processes has the potential to reduce the frequency of testing while increasing its quality, which may reduce clinicians’ cognitive workload and errors and enhance the adoption of EHRs and CDSSs.

Heuristic Evaluation of adverse drug reaction reporting System

“…In the study of Agharezaei on the examination of the laboratory information system, the highest numbers of problems were related to " exibility and e ciency of use" and the lowest was related to "assisting the user in error detection and recovery" [23] Harrington's study of self-management software for diabetes in 2022 Pareto analysis showed that 29 (57%) of the exploratory violations were related to "system-realworld compliance" and 8 (16%) to "aesthetic and simple design" [26].The results of studies Khajavi [22] and Nabouti[16], the component of " exibility and compatibility of the system" with the highest number of problems has been reported, which is in line with the results of the present study. Other study identi ed the mean severity scores range of exibility and e ciency of use base on heuristic evaluation on Clinical Decision Support System was 0.66, in which scores closer to 0 indicate a more usable system [27]. In the study of Ahmadian [28], the component of system adaptation to the real world and the aesthetic and simple design aspects had the most di culty in evaluating the radiology information system.…”

Formative evaluation of an emergency department clinical decision support system for agitation symptoms: a study protocol

“…We will first perform heuristic evaluation 55 of CDS interface usability before testing with clinician participants. This will consist of an expert team of three evaluators with experience in emergency medicine as well as CDS design who will navigate through different aspects of the CDS and judge the compliance and usability of the tool in agitation management triangulated between their expertise and usability standards.…”

“…We anticipate that our final product will be an EHR-integrated web application through a dedicated graphical Application Programming Interface. 54 Phase 2: usability testing of ED-TREAT Heuristic evaluation We will first perform heuristic evaluation 55 of CDS interface usability before testing with clinician participants. This will consist of an expert team of three evaluators with experience in emergency medicine as well as CDS design who will navigate through different aspects of the CDS and judge the compliance and usability of the tool in agitation management triangulated between their expertise and usability standards.…”

Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations

“…Usually, such systems are rule-based logical systems developed on pre-defined rules supplied by medical experts [10]. Despite the explainability offered by such systems, there are many challenges such as poor user experience and scarcity of involvement of medical experts in forming the knowledge base [12,20].…”

“…Examples of data-centric approaches are summarizing individual data instances (using common statistical methods like mean, mode, and variance), visualizing the data distribution to compare feature values of an instance to those across the remaining dataset and observing changes in model predictions through what-if analysis [18,19,22,57]. Additionally, data-centric explanations include data-driven rule-based approaches that are adopted commonly in medical DSS for assisting health experts [4,12,19,27,47].…”