Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects

Abstract

The view that the returns to educational investments are highest for early childhood interventions is widely held and stems primarily from several influential randomized trials-Abecedarian, Perry, and the Early Training Project-that point to super-normal returns to early interventions. This paper presents a de novo analysis of these experiments, focusing on two core issues that have received limited attention in previous analyses: treatment effect heterogeneity by gender and over-rejection of the null hypothesis due to multiple inference. To address the latter issue, I implement a statistical framework that combines summary index tests with Familywise Error Rate and False Discovery Rate corrections. The first technique reduces the number of tests conducted; the latter two adjust the p-values for multiple inference. The primary finding of the reanalysis is that girls garnered substantial short-and long-term benefits from the interventions. However, there were no significant long-term benefits for boys. These conclusions, which have appeared ambiguous when using "naive" estimators that fail to adjust for multiple testing, contribute to a growing literature on the emerging female-male academic achievement gap. They also demonstrate that in complex studies where multiple questions are asked of the same data set, it can be important to declare the family of tests under consideration and to either consolidate measures or report adjusted as well as unadjusted p-values.

Universal Child Care, Maternal Employment, and Children’s Long-Run Outcomes: Evidence from the US Lanham Act of 1940

“…Deming's (2009) paper, based on sibling fixed effects, finds that Head Start participants scored 0.23 standard deviations higher on a summary index of eight young adult outcomes. Positive long-run effects have also been estimated for participants in the Infant Health and Development, Perry, and Abecedarian programs (McCormick et al 2006;McCormick et al 2006;Anderson 2008;Heckman et al 2010;Conti, Heckman, and Pinto 2015).…”

“…Although several US jurisdictions offer universal preschool programs-including Georgia, Oklahoma, and Boston-they have not been operating long enough to shed light on whether the positive short-run impacts found in recent evaluations translate into long-run schooling and labor market success (Gormley and Gayer 2005;Fitzpatrick 2008;Cascio and Schanzenbach 2013;Weiland and Yoshikawa 2013). Conversely, evidence on the long-run impact of early childhood programs comes almost exclusively from highly targeted interventions, including the Head Start, Perry Preschool, and Abecedarian programs (e.g., Garces, Thomas, and Currie 2002;Ludwig and Miller 2007;Anderson 2008;Deming 2009). To my knowledge, the only evidence on the long-run impact of universal child care comes from a recent evaluation of the Norwegian system (Havnes and Mogstad 2011b).…”

Strengthening State Capabilities: The Role of Financial Incentives in the Call to Public Service*

“…These FDR-adjusted p-values account for the fact that we are testing multiple outcomes (e.g. Anderson (2008)). …”

Improving transparency in observational social science research: A pre-analysis plan approach

“…See Anderson (2008), Fink, McConnell, and Vollmer (2014), List, Shaikh, and Xu (2016), and Casey, Glennerster, and Miguel (2012) for further detail on multiple testing corrections.…”