Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review

Abstract

BackgroundComputerized decision support systems have raised a lot of hopes and expectations in the field of order entry. Although there are numerous studies reporting positive impacts, concerns are increasingly high about alert fatigue and effective impacts of these systems. One of the root causes of fatigue alert reported is the low clinical relevance of these alerts.ObjectiveThe objective of this systematic review was to assess the reported positive predictive value (PPV), as a proxy to clinical relevance, of decision support systems in computerized provider order entry (CPOE).MethodsA systematic search of the scientific literature published between February 2009 and March 2015 on CPOE, clinical decision support systems, and the predictive value associated with alert fatigue was conducted using PubMed database. Inclusion criteria were as follows: English language, full text available (free or pay for access), assessed medication, direct or indirect level of predictive value, sensitivity, or specificity. When possible with the information provided, PPV was calculated or evaluated.ResultsAdditive queries on PubMed retrieved 928 candidate papers. Of these, 376 were eligible based on abstract. Finally, 26 studies qualified for a full-text review, and 17 provided enough information for the study objectives. An additional 4 papers were added from the references of the reviewed papers. The results demonstrate massive variations in PPVs ranging from 8% to 83% according to the object of the decision support, with most results between 20% and 40%. The best results were observed when patients’ characteristics, such as comorbidity or laboratory test results, were taken into account. There was also an important variation in sensitivity, ranging from 38% to 91%.ConclusionsThere is increasing reporting of alerts override in CPOE decision support. Several causes are discussed in the literature, the most important one being the clinical relevance of alerts. In this paper, we tried to assess formally the clinical relevance of alerts, using a near-strong proxy, which is the PPV of alerts, or any way to express it such as the rate of true and false positive alerts. In doing this literature review, three inferences were drawn. First, very few papers report direct or enough indirect elements that support the use or the computation of PPV, which is a gold standard for all diagnostic tools in medicine and should be systematically reported for decision support. Second, the PPV varies a lot according to the typology of decision support, so that overall rates are not useful, but must be reported by the type of alert. Finally, in general, the PPVs are below or near 50%, which can be considered as very low.

Clinical Decision Support and Implications for the Clinician Burnout Crisis

“…They also proposed tiering of alerts to improve acceptance and consideration of human factors in CDS design [20] The analytic reviews analyzed the features of previously implemented CDS tools. Several reviews examined alert use, generally finding acceptance rates to be low [21] and override rates to be high [22][23][24]. Hussain et al delved into why alerts are so commonly overridden and found that although interruptive alerts are the most common design, they are also the least accepted.…”

“…Carli et al undertook a systematic review of positive predictive values (PPVs) of CDS alerts as a proxy for clinical relevance, finding massive PPV variations from 0% to 97%, with most in the 20%-40% range. PPVs were higher when the contextual information of the individual patient was considered in CDS, so the authors concluded that incorporating as much specific patient data as possible to improve PPV is more important than using a large database of knowledge that may generate many false positives and thus promote alert fatigue [24]. In a systematic review to understand CDS for drug allergies, Légat et al found that significant inaccuracies in the EHR due to the difficulty of structured documentation of allergies, the lack of standard allergy terminology, and irregular updating of allergy databases resulted in poor specificity and overall performance of CDS allergy alerts, along with override rates of up to 90%.…”

“…Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated. [20,24,97] -Role tailoring of alerts [21,82] -Tiering of alerts [20,80] -Iterative design [25] Harmful -Interruptive or hard stop alerts [21,25,80] -Lack of "right information" [18] -Lack of relevance [80,97,108] -Lack of integration [78]…”

Is the Climb Worth the View? The Savings/Alert Ratio for Reducing Vitamin D Testing

“…This pattern commonly occurs, with a recent review reporting the "positive predictive value" of such alerts ranging from 8 to 83%. 24 We considered the following five potential explanations in our setting:…”

Applying population health approaches to improve safe anticoagulant use in the outpatient setting: the DOAC Dashboard multi-cohort implementation evaluation study protocol

“…EHR alerts and dosing guidance might encourage appropriate dosing. However, alert fatigue has led many health systems to either deactivate electronic health record (EHR) drug alerts or display only the highest-risk alerts [18,19]. Additionally, many of the DOAC dosing issues may develop after the initial prescription (e.g., progressive renal function decline, failure to change VTE lead-in dosing for apixaban or rivaroxaban after 7 or 21 days, respectively) [20,21].…”