Accuracy of emergency physicians’ self-estimates of CT scan utilization and its potential effect on an audit and feedback intervention: a randomized trial

Abstract

Background Audit and feedback (A&F) has been used as a strategy to modify clinician behavior with moderate success. Although A&F is theorized to work by improving the accuracy of clinicians’ estimates of their own behavior, few interventions have included assessment of clinicians’ estimates at baseline to examine whether they account for intervention success or failure. We tested an A&F intervention to reduce computed tomography (CT) ordering by emergency physicians, while also examining the physicians’ baseline estimates of their own behavior compared to peers. Methods Our study was a prospective, multi-site, 20-month, randomized trial to examine the effect of an A&F intervention on CT ordering rates, overall and by test subtype. From the electronic health record, we obtained 12 months of baseline CT ordering per 100 patients treated for every physician from four emergency departments. Those who were randomized to receive A&F were shown a de-identified graph of the group’s baseline CT utilization, asked to estimate wherein the distribution of their own CT order practices fell, and then shown their actual performance. All participants also received a brief educational intervention. CT ordering rates were collected for all physicians for 6 months after the intervention. Pre-post ordering rates were compared using independent and repeated measures t tests. Results Fifty-one of 52 eligible physicians participated. The mean CT ordering rate increased significantly in both experimental conditions after the intervention (intervention pre = 35.7, post = 40.3, t = 4.13, p < 0.001; control pre = 33.9, post = 38.9, t = 3.94, p = 0.001), with no significant between-group difference observed at follow-up (t = 0.43, p = 0.67). Within the intervention group, physicians had poor accuracy in estimating their own ordering behavior at baseline: most overestimated and all guessed that they were in the upper half of the distribution of their peers. CT ordering increased regardless of self-estimate accuracy. Conclusions Our A&F intervention failed to reduce physician CT ordering: our feedback to the physicians showed most of them that they had overestimated their CT ordering behavior, and they were therefore unlikely to reduce it as a result. After “audit,” it may be prudent to assess baseline clinician awareness of behavior before moving toward a feedback intervention.

A Systematic Review of Electronic Medical Record Driven Quality Measurement and Feedback Systems

“…Both Hester et al, 2019 [37] and Dowling et al, 2020 [38] aimed to reduce low-value bronchiolitis management in pediatric care. Other similarities were found between studies that aimed to improve; pain management [39,40], prescribing practices [40][41][42][43], quality of discharge [41,42,44] and unnecessary test ordering [45,46]. The remaining studies had unique aims such as adherence to pneumonia guidelines [47], reducing heart failure re-admissions [48], improving lung-protective ventilation strategies [49], improving blood pressure control [50], and improving quality of glioma care [51,52].…”

“…All three ITS studies, and two CRCTs showed statistically significant improvement. In the two studies reporting null or negative effect, one reported no effect on blood pressure control [50] and the other reported a negative effect, where computed tomography (CT) orders increased significantly in both the intervention control groups and there was no significant difference between groups [45]. Nine of the studies reported secondary outcomes, of which most reported a positive effect (n = 8).…”