Using Heuristic Evaluation to Improve Sepsis Alert Usability

Pertiwi, Ariani Arista Putri; Fraczkowski, Daniel; Stogis, Sheryl L.; Lopez, Karen Dunn

doi:10.1016/j.cnc.2018.02.011

Cited by 16 publications

(6 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work should include a reevaluation of the CDS intervention after the recommendations against the heuristic violations have been implemented. In summary, heuristic evaluations have the potential to clarify usability issues within EHRs and CDSSs, not only after deployment but also before deployment, since they can be employed throughout various stages of system development [ 56 ]. Thus, this study reveals the value of including expert review methods at some point during the development process to ultimately achieve the goals of the system.…”

Section: Discussionmentioning

confidence: 99%

Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation

Cho¹,

Keenan²,

Madandola³

et al. 2022

JMIR Hum Factors

Self Cite

View full text Add to dashboard Cite

Background Poor usability is a primary cause of unintended consequences related to the use of electronic health record (EHR) systems, which negatively impacts patient safety. Due to the cost and time needed to carry out iterative evaluations, many EHR components, such as clinical decision support systems (CDSSs), have not undergone rigorous usability testing prior to their deployment in clinical practice. Usability testing in the predeployment phase is crucial to eliminating usability issues and preventing costly fixes that will be needed if these issues are found after the system’s implementation. Objective This study presents an example application of a systematic evaluation method that uses clinician experts with human-computer interaction (HCI) expertise to evaluate the usability of an electronic clinical decision support (CDS) intervention prior to its deployment in a randomized controlled trial. Methods We invited 6 HCI experts to participate in a heuristic evaluation of our CDS intervention. Each expert was asked to independently explore the intervention at least twice. After completing the assigned tasks using patient scenarios, each expert completed a heuristic evaluation checklist developed by Bright et al based on Nielsen’s 10 heuristics. The experts also rated the overall severity of each identified heuristic violation on a scale of 0 to 4, where 0 indicates no problems and 4 indicates a usability catastrophe. Data from the experts’ coded comments were synthesized, and the severity of each identified usability heuristic was analyzed. Results The 6 HCI experts included professionals from the fields of nursing (n=4), pharmaceutical science (n=1), and systems engineering (n=1). The mean overall severity scores of the identified heuristic violations ranged from 0.66 (flexibility and efficiency of use) to 2.00 (user control and freedom and error prevention), in which scores closer to 0 indicate a more usable system. The heuristic principle user control and freedom was identified as the most in need of refinement and, particularly by nonnursing HCI experts, considered as having major usability problems. In response to the heuristic match between system and the real world, the experts pointed to the reversed direction of our system’s pain scale scores (1=severe pain) compared to those commonly used in clinical practice (typically 1=mild pain); although this was identified as a minor usability problem, its refinement was repeatedly emphasized by nursing HCI experts. Conclusions Our heuristic evaluation process is simple and systematic and can be used at multiple stages of system development to reduce the time and cost needed to establish the usability of a system before its widespread implementation. Furthermore, heuristic evaluations can help organizations develop transparent reporting protocols for usability, as required by Title IV of the 21st Century Cures Act. Testing of EHRs and CDSSs by clinicians with HCI expertise in heuristic evaluation processes has the potential to reduce the frequency of testing while increasing its quality, which may reduce clinicians’ cognitive workload and errors and enhance the adoption of EHRs and CDSSs.

show abstract

Section: Discussionmentioning

confidence: 99%

Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation

Cho¹,

Keenan²,

Madandola³

et al. 2022

JMIR Hum Factors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Descriptions of discount usability in EHR system evaluations generally focus on either heuristic evaluations, [18][19][20] a specific feature under test, 21,22 or both. 19,23 We are aware of no descriptions in the literature of the broad application of sitespecific, discount user testing of a health care system's local configuration, despite the potential for such configurations to substantially impact usability. This case report highlights the value of user testing using principles of discount usability at MU Health Care.…”

Section: Discussionmentioning

confidence: 99%

Improving the User Experience with Discount Site-Specific User Testing

et al. 2022

View full text Add to dashboard Cite

Objectives Poor electronic health record (EHR) usability is associated with patient safety concerns, user dissatisfaction, and provider burnout. EHR certification requires vendors to perform user testing. However, there are no such requirements for site-specific implementations. Health care organizations customize EHR implementations, potentially introducing usability problems. Site-specific usability evaluations may help to identify these concerns, and “discount” usability methods afford health systems a means of doing so even without dedicated usability specialists. This report characterizes a site-specific discount user testing program launched at an academic medical center. We describe lessons learned and highlight three of the EHR features in detail to demonstrate the impact of testing on implementation decisions and on users. Methods Thirteen new EHR features which had already undergone heuristic evaluation and iterative design were evaluated over the course of three user test events. Each event included five to six users. Participants used think aloud technique. Measures of user efficiency, effectiveness, and satisfaction were collected. Usability concerns were characterized by the type of usability heuristic violated and by correctability. Results Usability concerns occurred at a rate of 2.5 per feature tested. Seventy percent of the usability concerns were deemed correctable prior to implementation. The first highlighted feature was moved to production despite low single ease question (SEQ) scores which may have predicted its subsequent withdrawal from production based on post implementation feedback. Another feature was rebuilt based on usability findings, and a new version was retested and moved to production. A third feature highlights an easily correctable usability concern identified in user testing. Quantitative usability metrics generally reinforced qualitative findings. Conclusion Simplified user testing with a limited number of participants identifies correctable usability concerns, even after heuristic evaluation. Our discount usability approach to site-specific usability has a role in implementations and may improve the usability of the EHR for the end user.

show abstract

“…Second, if the electronic monitoring and alerting system is difficult to use and/or is not regarded as helpful to clinicians, it can lead to confusion, to frustration, and possibly to worse patient care. 29 For example, if the alert physicians receive contains too little information (or too much), or if the action required is not clear, physicians may find the system too difficult or burdensome to use. 29 To improve system usability, input from clinicians was solicited in some studies, followed by adaptations.…”

Section: Discussionmentioning

confidence: 99%

“… 29 For example, if the alert physicians receive contains too little information (or too much), or if the action required is not clear, physicians may find the system too difficult or burdensome to use. 29 To improve system usability, input from clinicians was solicited in some studies, followed by adaptations. Examples of clinician-driven adaptations included allowing a nurse to “snooze” an alert for 6 hours if the patient is already under assessment for sepsis, or implementing a “traffic light” system on a dashboard to visually show clinicians which patients are in a warning zone (yellow) or need urgent attention (red).…”

Section: Discussionmentioning

confidence: 99%

The Use of Patient Monitoring Systems to Improve Sepsis Recognition and Outcomes: A Systematic Review

Gale

Hall²

2020

J Patient Saf

View full text Add to dashboard Cite

Introduction The aim of this systematic review was to determine the impact of automated patient monitoring systems (PMSs) on sepsis recognition and outcomes. Methods Systematic searches were conducted using CINAHL, MEDLINE, and Cochrane, for articles published from 2008 through 2018. English-language, peer-reviewed articles that reported the impact of PMS on sepsis care were included. For selected articles, the authors abstracted information, with the study designed to be compliant with Preferred Reporting Items for Systematic reviews and Meta-Analyses guidelines. Results Nineteen articles were identified for inclusion: 4 systematic reviews and 15 individual studies. Study design and quality varied, with some randomized controlled trials and quasiexperimental studies, as well as many observational studies. Study results for outcome measures (e.g., mortality, intensive care unit [ICU] length of stay, ICU transfer) were mixed, with more than half of the studies showing a significant improvement in at least one measure. Evidence for process measure (e.g., time to antibiotic administration, lactate measurement, etc.) improvement was of moderate strength across multiple types of hospital units, and evidence was most consistent outside the ICU. Conclusions Automated sepsis PMSs have the potential to improve sepsis recognition and outcomes, but current evidence is mixed on their effectiveness. More high-quality studies are needed to understand the effects of PMSs on important sepsis-related process and outcome measures in different hospital units.

show abstract

Using Heuristic Evaluation to Improve Sepsis Alert Usability

Cited by 16 publications

References 32 publications

Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation

Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation

Improving the User Experience with Discount Site-Specific User Testing

The Use of Patient Monitoring Systems to Improve Sepsis Recognition and Outcomes: A Systematic Review

Contact Info

Product

Resources

About