Evaluation of statistical methods used in the analysis of interrupted time series studies: a simulation study

Abstract

Interrupted time series (ITS) studies are frequently used to evaluate the effects of population level interventions or exposures. To our knowledge, no studies have compared the performance of different statistical methods for this design. We simulated data to compare the performance of a set of statistical methods under a range of scenarios which included different level and slope changes, varying lengths of series and magnitudes of autocorrelation. We also examined the performance of the Durbin-Watson (DW) test for detecting autocorrelation. All methods yielded unbiased estimates of the level and slope changes over all scenarios. The magnitude of autocorrelation was underestimated by all methods, however, restricted maximum likelihood (REML) yielded the least biased estimates. Underestimation of autocorrelation led to standard errors that were too small and coverage less than the nominal 95%. All methods performed better with longer time series, except for ordinary least squares (OLS) in the presence of autocorrelation and Newey-West for high values of autocorrelation. The DW test for the presence of autocorrelation performed poorly except for long series and large autocorrelation. From the methods evaluated, OLS was the preferred method in series with fewer than 12 points, while in longer series, REML was preferred. The DW test should not be relied upon to detect autocorrelation, except when the series is long. Care is needed when interpreting results from all methods, given confidence intervals will generally be too narrow. Further research is required to develop better performing methods for ITS, especially for short series.

Comparison of Six Statistical Methods for Interrupted time Series Studies: Empirical Evaluation of 190 Published Series

“…For longer series, as expected, the confidence intervals are narrow, with many excluding no and negative autocorrelation estimates. for autocorrelation (which OLS does not), and in a numerical simulation study investigating the performance of these methods, PW was shown to perform better than OLS for data series approximately longer than 24 points (12). The results in our empirical investigation therefore likely reflect the influence of shorter data series.…”

“…< 100 data points), with the ARIMA and OLS autocorrelation estimates being substantially smaller than REML. Given the true underlying autocorrelation would not be expected to vary by series length, the stability of the REML estimates over the different series lengths is suggestive of it being the preferable estimator, which has been shown in numerical simulation studies to be the case (12,26).…”

“…Six statistical methods were used to analyse the ITS datasets assuming first order autocorrelation (lag-1) ( Table 1). The methods were chosen because they have commonly been used in practice (8)(9)(10)(11) or because of they have been shown (through numerical simulation) to have improved performance (12). The methods were:…”

“…Turner et al undertook a statistical simulation study examining the performance of statistical methods for analysing ITS data, where the methods were those commonly used in practice or had shown potential to perform well (12). This simulation study provided insight into how these statistical methods performed under different scenarios, including different level and slope changes, varying magnitudes of underlying autocorrelation and series lengths.…”

“…To our knowledge, there has been no study that has empirically compared different methods for analysing ITS data when applied to a large sample of real-world data sets. We therefore undertook such an evaluation, where we aimed to compare level and slope change estimates, their standard errors, confidence intervals and pvalues, and estimates of autocorrelation, obtained from the set of statistical methods used in the Turner et al simulation study (12).…”