Clinical decision support alert malfunctions: analysis and empirically derived taxonomy

Abstract

ObjectiveTo develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions.Materials and MethodsWe identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions.ResultsWe analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common.DiscussionAcross organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS.ConclusionCDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Artificial intelligence‐based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance

“…In order for a CDSS to fit the workflow of a particular clinic, customization of the CDSS might be necessary. Therefore, the customization functionality offered by each CDSS should be taken into account during selection . Another consideration related to the local workflow is whether all the necessary data for the proper functioning of the CDSS is available in that specific point in the workflow …”

“…To facilitate the discovery of CDSS malfunctions, mechanisms need to be in place for receiving user feedback and acting on it . Besides, CDSS malfunctions can be identified by a combination of qualitative and quantitative analyses (e.g., of firing rates for alert systems or overrides for recommender CDSS) . Visual detection and statistical process control analysis have shown good results as tools to detect malfunction .…”

Economic impact of clinical decision support interventions based on electronic health records

“…[55] These may also lead to yet another cost-driver and possibly cause unpredictable economic damage. [56,57] Transferability for other countries All included studies where based on cost data and trials from the United States or Canada. Consequently, current research progress on the economic potentials of EHR based CDS systems on rising healthcare expenditure in Europe and worldwide cannot be derived.…”

A Quality Improvement Initiative to Decrease Platelet Ordering Errors and a Proposed Model for Evaluating Clinical Decision Support Effectiveness

“…43 Given that these types of errors can be hard to predict, organizations should utilize usability testing and postimplementation monitoring to mitigate the risk of unintended errors. 26,42,[44][45][46] The Safety Assurance Factors for Electronic Health Record Resilience guidelines include usability testing and monitoring as recommended practices for institutions using CDS and CPOE. 47 Institutions using CDS should have procedures that require regular review of order sets that includes both a clinical evaluation and patient safety evaluation with data when possible.…”