Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

Abstract

Objectives Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results We find important variability in the estimated treatment effect, and a large degree of imbalance in 'unobserved' covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

Heterogeneous effects of adolescent violent victimization on problematic outcomes in early adulthood*

“…Rosenbaum bounds indicate how large an unobserved confounding variable's effect would need to be on adolescent violent victimization to render the observed effects null. Although these bounds are not perfectly suited to the distributions of all the outcome variables assessed here, their calculation is recommended to provide evidence of robustness (Caliendo & Kopeinig, ; Loughran et al, ).…”

“…Unlike in standard regression approaches, when it comes to model building in propensity score matching, “parsimony is not necessarily a virtue” (Apel & Sweeten, : 559). Extensive covariates are necessary to capture—as best as possible—nonrandom sources of selection into victimization (Loughran, Wilson, Nagin, & Piquero, ).…”

Juvenile Waiver as a Mechanism of Social Stratification: A Focus on Human Capital*

“…For one‐to‐one matching, we observed Γ = 1.61, which can be interpreted as an unobserved covariate that would need to be 61 percent more likely in the transfer group and a strong predictor of the outcome to render the treatment effect insignificant at α = .05. Given the wide range of confounders we eliminated, this statistic suggests that such an unmeasured confounder would need to be large to serve as a form of hidden bias, and it further ensures the robustness of our results (Loughran et al., ; Rosenbaum, ).…”

“…The propensity score represents the probability that an individual receives some treatment (i.e., transfer) conditional on a vector of observed covariates (Rosenbaum, ) and can be used to create matched sets over which all observed confounders are balanced. Given the richness of the data considered, we were able to eliminate a wide range of potential confounders, giving us a high degree of confidence in the comparability of matched groups (Loughran et al., ). Nevertheless, we still considered the sensitivity of the estimated treatment effects to potential hidden biases.…”

When Frontloading Backfires: Exploring the Impact of Outsourcing Correctional Interventions on Mechanisms of Social Control

“…This bias may, in turn, cause the conditional independence assumption to fail (Rosenbaum 2005). While the assumption of unconfoundedness is not testable, conducting sensitivity analyses can ascertain the magnitude of hidden bias that would have to be present to reverse the conclusions of a study that used a propensity scoring and matching approach (Rosenbaum 2002;Loughran et al 2015). 5 To present our results economically, we limit our description of our sensitivity findings to the eighteen-month follow-up effects for the macro-level analysis: parolees frontloaded residential community-based treatment services compared to those not receiving those services.…”