Evaluating Kindergarten Retention Policy

Abstract

This article considers the policy of retaining low-achieving children in kindergarten rather than promoting them to first grade. Under the stable unit treatment value assumption (SUTVA) as articulated by Rubin, each child at risk of retention has two potential outcomes: Y(1) if retained and Y(0) if promoted. But SUTVA is questionable, because a child's potential outcomes will plausibly depend on which school that child attends and also on treatment assignments of other children. We develop a causal model that allows school assignment and peer treatments to affect potential outcomes. We impose an identifying assumption that peer effects can be summarized through a scalar function of the vector of treatment assignments in a school. Using a large, nationally representative sample, we then estimate (1) the effect of being retained in kindergarten rather than being promoted to the first grade in schools having a low retention rate, (2) the retention effect in schools having a high retention rate, and (3) the effect of being promoted in a low-retention school as compared to being promoted in a high-retention school. This third effect is not definable under SUTVA. We use multilevel propensity score stratification to approximate a two-stage experiment. At the first stage, intact schools are blocked on covariates and then, within blocks, randomly assigned to a policy of retaining comparatively more or fewer children in kindergarten. At the second stage, "at-risk" students within schools are blocked on covariates and then assigned at random to be retained. We find evidence that retainees learned less on average than did similar children who were promoted, a result found in both high-retention and low-retention schools. We do not detect a peer treatment effect on low-risk students.

Assessing the Effectiveness of Quantway®: A Multilevel Model With Propensity Score Matching

“…One of the study's objectives was to examine variation in performance across classrooms and institutions to see if and where improvement efforts should be targeted. With the HLM model, we could capture not only the average effect of Quantway 1 but also variability in its effect for each level (Hong & Raudenbush, 2005, 2006Raudenbush & Bryk, 2002). In the Level 1 model below, η ijkl is the log-odds of success or enrollment, and QW ijkl is a dummy variable indicating whether the outcome was from the Quantway 1 group (coded as 1) or the matched comparison group (coded as 0).…”

“…In other words, all relevant covariates were included in the propensity score analysis, so that the bias due to unmeasured covariates could be ignored. Thus, we examined the sensitivity of the estimated Quantway 1 effects to possible unmeasured confounders (Hong & Raudenbush, 2005, 2006Lin, Psaty, & Kronmal, 1998). Given some unmeasured covariates (U), the Quantway 1 effect (δ) can be reestimated by adjusting for hypothesized hidden bias {γ[E(…”

“…In the Level 1 model below, η ijkl is the log-odds of success or enrollment, and QW ijkl is a dummy variable indicating whether the outcome was from the Quantway 1 group (coded as 1) or the matched comparison group (coded as 0). We included individual propensity scores, PS ijkl , as a further safeguard to remove any potential remaining bias (Hong & Raudenbush, 2005, 2006 The Level 1 Model (measurement) is formally expressed as: η ijkl = π 0jkl + π 1jkl (QW ijkl ) + π 2jkl (PS ijkl ). The Level 1 model is in essence a measurement model for the individual effect of Quantway 1 on each separate student (Raudenbush & Bryk, 2002).…”

“…As Quantway 1 was designed not only to get students through their developmental mathematics sequences but also to prepare them to meet their college mathematics requirements, this analysis was particularly important in determining Quantway 1's effectiveness. We applied an HLM approach to all of these outcome analyses because potential outcomes most likely depended on the institutions and classrooms to which students belong (Hong & Raudenbush, 2005, 2006.…”

“…First, we conducted propensity score matching to identify a group of students who followed traditional developmental mathematics sequences with similar characteristics to Quantway 1 students. To obtain propensity scores, we took an HLM approach (Hong & Raudenbush, 2005, 2006Raudenbush & Bryk, 2002;Yamada & Bryk, 2016) and constructed a two-level model with a total of 37 student-level covariates including student background characteristics and prior course taking and success patterns during the 2 years prior to the Quantway 1 term (see Table 1). We estimated model parameters by leveraging maximum likelihood via penalized quasi-likelihood estimation using HLM 7 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011).…”