Testing the Validity of the Single Interrupted Time Series Design

Abstract

Given the complex relationships between patients' demographics, underlying health needs, and outcomes, establishing the causal effects of health policy and delivery interventions on health outcomes is often empirically challenging. The single interrupted time series (SITS) design has become a popular evaluation method in contexts where a randomized controlled trial is not feasible. In this paper, we formalize the structure and assumptions underlying the single ITS design and show that it is significantly more vulnerable to confounding than is often acknowledged and, as a result, can produce misleading results. We illustrate this empirically using the Oregon Health Insurance Experiment, showing that an evaluation using a single interrupted time series design instead of the randomized controlled trial would have produced large and statistically significant results of the wrong sign. We discuss the pitfalls of the SITS design, and suggest circumstances in which it is and is not likely to be reliable.

Association Between Filgrastim Biosimilar Availability and Changes in Claim Payments and Patient Out-of-Pocket Costs for Biologic Filgrastim Products

“…Given the expected lag between the launch of a new drug and its adoption, 22 we empirically determined the lag time between biosimilar filgrastim market entry and its utilization. Because of its demonstrated utility in the econometrics and interrupted time series (ITS) literature, 23,24 we used a structural break test of unknown break point with a sup-Wald statistic, as described by Piehl et al 23 and more recently by Baicker and Svoronos 2019, 24 to determine the change point in branded filgrastim utilization for use in our main ITS analysis. Then, we used an ITS design with ordinary least squares regression to estimate the impact of biosimilar filgrastim market entry on the claims payments and patient OOP costs for branded filgrastim.…”

“…After biosimilar filgrastim market entry, there was a rapid and significant decrease in branded filgrastim utilization, which was owing to a rapid and significant uptake of biosimilar filgrastim. Using an econometrics method, 24 we quantified this break in utilization and found that significant uptake of biosimilar filgrastim began in February 2016. This corroborates previously published data on filgrastim-sndz uptake in Medicare 14,15 and commercial 14 populations, as well as previous research on drug adoption, which reported a typical delay of several months Table 1.…”

“…This supports the use of ITS in this setting. 24 Therefore, despite that biosimilar filgrastim was launched in September 2015, our preintervention period was from January 1, 2014, to February 29, 2016, and the postintervention period was from March 1, 2016, to December 31, 2017.…”

“…This allowed us to determine that ITS was a suitable estimation method for our question and maximized our signal-tonoise ratio by identifying the best intervention point for the purposes of ITS. 24 Second, this study is novel in that it is the first, we believe, to empirically estimate the effect of market entry of a new biosimilar on insurer payments and patient out-of-pocket costs in the United States. Third, this study assessed insurer payments and patient out-of-pocket costs by health plan type.…”

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

“…For example, Baicker and Svoronos recently reanalysed data from the Oregon Health Insurance experiment using SITS and found results that were not consistent with the randomised trial findings. 30 Therefore, it is generally recommended to add a control group to ITS wherever possible. 10 22 Bernal and colleagues' study reviewed various types of controls that can be added to strengthen ITS design, including location and characteristic-based controls and control outcome.…”