Short Comparative Interrupted Time Series Using Aggregate School-Level Data in Education Research

Abstract

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on a variety of assumptions and design decisions that are not clearly outlined in the literature. This article aims to serve as a practice guide for applied researchers when deciding how and whether to use this approach. We begin by providing an overview of the assumptions needed to estimate causal effects using school-level data, common threats to validity faced in practice and what effects can and cannot be estimated using school-level data. We then examine two analytic decisions researchers face in practice when implementing the design: correctly modeling the pretreatment functional form, which is modeling the preintervention trend, and selecting comparison cases. We then illustrate the use of this design in practice drawing on data from the implementation of the school improvement grant (SIG) program in Ohio. We conclude with advice for applied researchers implementing this design.

Is “Whole Child” Education Obsolete? Public School Principals’ Educational Goal Priorities in the Era of Accountability

“…Here, we employ comparative interrupted time series (CITS) design, a quasi-experimental research method to explore the policy impact with controls for observed and unobserved differences between states and common time trends (Hallberg et al, 2018). CITS allows us to use repeated measures of principals’ goal priorities and compare pre-NCLB and post-NCLB trends between the two groups of states.…”

The Relationship Between School Turnaround and Student Outcomes: A Meta-Analysis

“…Studies that do not separate the results by a specific turnaround model or specify which turnaround model was implemented but otherwise followed our definition of school turnaround are included in an inclusive category. For instance, if a study examines how school transformation and turnaround both influence student outcomes, then we would classify these estimates in the general category (i.e., Hallberg et al, 2018; Sun et al, 2017). To understand whether a change in management or governance is associated with improved student outcomes, we compare schools that maintained local control during their turnaround process to restart schools and schools that were taken over by the state (i.e., Recovery School District; Achievement School District).…”

Still Where, Not If, You’re Poor: International Baccalaureate Opportunities to Learn International-Mindedness and Proximity to U.S. Cities

“…Several options exist for matching comparison cases with treatment cases in education research, but the relatively small proportion of IB schools in the United States (i.e., <2% of all public schools) prompted us to use a nearest neighbor 1:1 match. That choice avoided computational and interpretive challenges that can come from highly unbalanced designs, which would have been likely had we matched each IB school against multiple comparison schools (Hallberg et al, 2018; Keppel & Wickens, 2004). That choice did not, meanwhile, thwart our identification of comparisons that best approximated our data set’s schools, which opted to offer IB (Hallberg et al, 2018; Jacob et al, 2016).…”

“…That choice avoided computational and interpretive challenges that can come from highly unbalanced designs, which would have been likely had we matched each IB school against multiple comparison schools (Hallberg et al, 2018; Keppel & Wickens, 2004). That choice did not, meanwhile, thwart our identification of comparisons that best approximated our data set’s schools, which opted to offer IB (Hallberg et al, 2018; Jacob et al, 2016). We expected observable factors, not randomness, to associate with whether a school adopted an IB program, so our matching approach adhered to logic from research on IB access that advocates for controlling the type of self-selection that can plagues IB studies (May et al, 2014).…”

“…We expected observable factors, not randomness, to associate with whether a school adopted an IB program, so our matching approach adhered to logic from research on IB access that advocates for controlling the type of self-selection that can plagues IB studies (May et al, 2014). Moreover, we aimed to facilitate associative rather than causal claims, so we deemed a nearest neighbor 1:1 match to be appropriate, recognizing a greater likelihood of identifying plausible comparison schools than selecting ideal matches (Hallberg et al, 2018; Stuart, 2007).…”