Formative Usability Testing Reduces Severe Blood Product Ordering Errors

Abstract

Background Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process. Objectives There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD. Methods A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t-tests. Results Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9–57%; p = 0.03). Time on task did not vary significantly. Conclusion UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.

Displaying Cost and Completion Time for Reference Laboratory Test Orders—A Randomized Controlled Trial

“…In Orenstein et al's study, existing blood product order sets were first modified by a multidisciplinary expert committee then later revised through extensive usability testing by providers through scenario-based feedback. 27 It was noted that persistent failure was detected after the expert modification but the adoption of user-centered design enabled a significant reduction in user errors. Accordingly, collection and integration of user feedback is critical to ensure success of CDS tools.…”

Clinician Perceptions on the Use of Free-Text Communication Orders

“…Substantial changes require vendors and healthcare systems to allocate resources to perform usability testing of medication ordering systems before, during, and after hospital EHR implementation to understand work systems, minimize workarounds and identify unintended consequences. 32,33…”

The Effect of Electronic Health Record Usability Redesign on Annual Screening Rates in an Ambulatory Setting

“…[18][19][20][21] Other studies of EHR usability focus on poor usability and its association with patient harm and clinician dissatisfaction. [2][3][4]6,22,23 While it is important to continue to describe the impact of poor usability, our understanding of usability is enhanced by careful descriptions of the improvements seen with corrections of usability problems. Ours is one of the first projects to demonstrate that UI redesign with the aim of correction of usability problems may be associated with improvements in quality of care as measured by clinical quality measures.…”