A reanalysis of cluster randomized trials showed interrupted time-series studies were valuable in health system evaluation

Fretheim, Atle; Zhang, Fang; Ross‐Degnan, Dennis; Oxman, Andrew D; Cheyne, Helen; Foy, Robbie; Goodacre, Steve; Herrin, Jeph; Kerse, Ngaire; McKinlay, R. James; Wright, Adam; Soumerai, Stephen B.

doi:10.1016/j.jclinepi.2014.10.003

Cited by 101 publications

(96 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[19][20][21] Other strengths of this study are the consistent collection of validated administrative data throughout the study period and the incorporation of a control hospital, which enabled the development of more valid counterfactual assumptions (what would likely have happened to key outcomes in the absence of the intervention). 18,19 Our data show increasing wait times and left-without-being-seen rates during the pre-intervention phase, suggesting that environmental, volume, or case mix factors negatively influenced ED performance during the February to August period. These negative trends reversed at both hospitals during the August to November period, suggesting that post-intervention changes in operational performance, in part, were related to seasonal or environmental factors, or regression to the norm.…”

Section: Discussionmentioning

confidence: 72%

“…ITS analysis accounts for longitudinal trends preceding and following the intervention 18 and is the strongest quasi-experimental approach to evaluating the effect of an intervention over time. [19][20][21] Other strengths of this study are the consistent collection of validated administrative data throughout the study period and the incorporation of a control hospital, which enabled the development of more valid counterfactual assumptions (what would likely have happened to key outcomes in the absence of the intervention).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Impact of physician payment mechanism on emergency department operational performance

Innes

Scheuermeyer

Marsden

et al. 2018

CJEM

View full text Add to dashboard Cite

CLINICIAN'S CAPSULEWhat is known about the topic? Fee-for-service compensation may motivate physicians to see more patients and improve throughput, or drive excessive testing and referral behaviour that undermine emergency performance. What did this study ask? Does fee-for-service payment reduce emergency wait times, length of stay, and left without being seen rates? What did this study find? We observed an unsustained 24% reduction in time to physician, but no change in length of stay or left without being seen rates. Why does this study matter to clinicians? Fee-for-service compensation may have transient motivational effect but is unlikely to generate system-level improvement. ABSTRACTObjective: Fee-for-service payment may motivate physicians to see more patients and achieve higher productivity. In 2015, emergency physicians at one Vancouver hospital switched to fee-for-service payment, while those at a sister hospital remained on contract, creating a natural experiment where the compensation method changed, but other factors remained constant. Our hypothesis was that fee-for-service payment would increase physician efficiency and reduce patient wait times. Methods: This interrupted time series with concurrent control analysed emergency department (ED) performance during a 42-week period, encompassing the intervention (fee for service). Data were aggregated by week and plotted in a time series fashion. We adjusted for autocorrelation and developed general linear regression models to assess level and trend changes. Our primary outcome was the wait time to physician. Results: Data from 142,361 ED visits were analysed. Baseline wait times rose at both sites during the pre-intervention phase.Immediately post-intervention, the median wait time increased by 2.4 minutes at the control site and fell by 7.2 minutes at the intervention site (difference = 9.6 minutes; 95% confidence interval, 2.9-16.4; p = 0.007). The wait time trend (slope) subsequently deteriorated by 0.5 minutes per week at the intervention site relative to the expected counterfactual (p for the trend difference = 0.07). By the end of the study, cross-site differences had not changed significantly from baseline. Conclusion: Fee-for-service payment was associated with a 9.6-minute (24%) reduction in wait time, compatible with an extrinsic motivational effect; however, this was not sustained, and the intervention had no impact on other operational parameters studied. Physician compensation is an important policy issue but may not be a primary determinant of ED operational efficiency. RÉSUMÉObjectif: Le mode de paiement à l'acte peut inciter les médecins à voir plus de patients et ainsi à accroître la productivité. En 2015, des urgentologues travaillant dans un hôpital de Vancouver sont passés au mode de paiement à l'acte, tandis que d'autres travaillant dans un hôpital de type et de grosseur comparables ont conservé le mode de paiement contractuel, ce qui a permis de mener une expérience naturelle dans laquelle seuls les modes de rémunération avaient c...

show abstract

Section: Discussionmentioning

confidence: 72%

Section: Discussionmentioning

confidence: 99%

Impact of physician payment mechanism on emergency department operational performance

Innes

Scheuermeyer

Marsden

et al. 2018

CJEM

View full text Add to dashboard Cite

show abstract

“…As these 34 physicians received the results of the cluster-randomized trial, it is possible that this may have increased antiemetic prescription during the before-after study. Nonetheless, a large, risk-dependent change in decision making with a corresponding risk-dependent change in patient outcome makes it quite plausible that the observed difference between the before and after periods is caused by the intervention [36]. From a study design perspective, a prediction model impact study should be regarded as a program evaluation, in which the implementation of a complex intervention is studied [37].…”

Section: Handling Missing Predictor Values When Using the Modelmentioning

confidence: 99%

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

et al. 2018

View full text Add to dashboard Cite

An important aim of clinical prediction models is to positively impact clinical decision making and subsequent patient outcomes. The impact on clinical decision making and patient outcome can be quantified in prospective comparative-ideally cluster-randomized-studies, known as 'impact studies'. However, such impact studies often require a lot of time and resources, especially when they are (cluster-)randomized studies. Before envisioning such large-scale randomized impact study, it is important to ensure a reasonable chance that the use of the prediction model by the targeted healthcare professionals and patients will indeed have a positive effect on both decision making and subsequent outcomes. We recently performed two differently designed, prospective impact studies on a clinical prediction model to be used in surgical patients. Both studies taught us new valuable lessons on several aspects of prediction model impact studies, and which considerations may guide researchers in their decision to conduct a prospective comparative impact study. We provide considerations on how to prepare a prediction model for implementation in practice, how to present the model predictions, and how to choose the proper design for a prediction model impact study.

show abstract

“…72 Fretheim and colleagues suggest adding time series approaches to the overall comparison of randomised groups, so as to gauge changes in effect of the intervention over time.…”

Section: Study Analysismentioning

confidence: 99%

A brief history of the cluster randomised trial design

Moberg

Kramer

2015

J R Soc Med

View full text Add to dashboard Cite

A reanalysis of cluster randomized trials showed interrupted time-series studies were valuable in health system evaluation

Abstract: The ITS design is valuable for evaluation of health systems interventions, both when RCTs are not feasible and in the analysis and interpretation of data from C-RCTs.

Cited by 101 publications

References 22 publications

Impact of physician payment mechanism on emergency department operational performance

Impact of physician payment mechanism on emergency department operational performance

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

A brief history of the cluster randomised trial design

Contact Info

Product

Resources

About