Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes?

Abstract

For RCTs of education interventions, it is often of interest to estimate associations between student and mediating teacher practice outcomes, to examine the extent to which the study's conceptual model is supported by the data, and to identify specific mediators that are most associated with student learning. This paper develops statistical power formulas for such exploratory analyses under clustered school-based RCTs using ordinary least squares (OLS) and instrumental variable (IV) estimators U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance John Q. Easton Acting Commissioner October 2009This report is in the public domain. While permission to reprint this publication is not necessary, the citation should be: Disclosure of Potential Conflicts of InterestThe author for this report, Dr. Peter Z. Schochet, is an employee of Mathematica Policy Research with whom IES contracted to develop the methods that are presented in this report. Dr. Schochet and other MPR staff do not have financial interests that could be affected by the content in this report. Disclosure of Potential Conflicts of Interest iii ForewordThe National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased large-scale evaluations of education programs and practices supported by federal funds; provides research-based technical assistance to educators and policymakers; and supports the synthesis and the widespread dissemination of the results of research and evaluation throughout the United States.In support of this mission, NCEE promotes methodological advancement in the field of education evaluation through investigations involving analyses using existing data sets and explorations of applications of new technical methods, including cost-effectiveness of alternative evaluation strategies. The results of these methodological investigations are published as commissioned, peer reviewed papers, under the series title, Technical Methods Reports, posted on the NCEE website at http://ies.ed.gov/ncee/pubs/. These reports are specifically designed for use by researchers, methodologists, and evaluation specialists. The reports address current methodological questions and offer guidance to resolving or advancing the application of highquality evaluation methods in varying educational contexts.This NCEE Technical Methods paper addresses whether typical large-scale RCT designs have sufficient statistical power for meditational analyses that associate teacher practice and student achievement outcomes. These exploratory analyses are important for helping to understand key pathways through which the intervention affects student learning as hypothesized by the study's conceptual model, and for identifying specific teacher practices that are most associated with student learning. These analyses, however, will be informative only if the study has sufficient statistical power for estimating mediator-achievement...

Croon’s Bias-Corrected Factor Score Path Analysis for Small- to Moderate-Sample Multilevel Structural Equation Models

“…Yet, substantive literature has emphasized that the benefits of studying theories, interventions, and intermediate processes are not limited to large-scale studies—small- to moderate-scale studies offer critical contributions to theory and social issues when they are well executed (e.g., Bodner & Bliese, 2018; Walton, 2014). In many areas of research, samples of fewer than 80 organizations are typical and samples greater than 80 may be considered prohibitively large (e.g., Schochet, 2011; Spybrook, Shi, & Kelcey, 2016). Perhaps because of the mismatch between the scale of many multilevel studies and the large-scale requirements of ML estimation in MLSEM, literature reviews have reported a widespread absence in the appropriate adjustment for measurement error (e.g., Aguinis, Edwards, & Bradley, 2017).…”

Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement

“…There are several approaches to assessing mediation, including use of instrumental variables, (Bloom 2005a), and regression-based approaches that assess the relationship between the program and outcome through an indirect “mediated” path, and a direct effect from program to outcome (Baron and Kenny, 1986; Krull and MacKinnon, 2001). Both methods share the problem of having limited statistical power, and this is especially problematic if the mediator is at the cluster level (Schochet, 2009). This is the case of the iRAISE evaluation with teachers (clusters) randomized, instruction being the mediator, and student achievement the distal outcome.…”

Statistical Power for Causally Defined Indirect Effects in Group-Randomized Trials With Individual-Level Mediators

“…There is a growing recognition of the critical contributions provided by study designs that facilitate inferences concerning both the total and indirect effects of a treatment (Institute of Educational Sciences, U.S. Department of Education, & National Science Foundation, 2013; Schochet, 2011; Sobel, 2008). Designs that facilitate both types of inferences potentially offer a more holistic and richer description of the programs under study and their guiding theories because they can complement “what works” type questions with the comprehensive study of the specific causal connections implied by program theories (Raudenbush & Sadoff, 2008).…”

“…When data are structured hierarchically, a prominent design to facilitate inferences concerning the total and indirect effects is the group-randomized trial (e.g., Schochet, 2011). In this design, groups are randomly assigned to treatment conditions to assess the treatment's impact on an outcome and, when a mediator is observed, to probe the extent to which this impact flows through a mediating or intermediate variable implied by theory.…”

“…We delineate the sensitivity of group-randomized designs to concurrently detect the chain of relationships that link a treatment with improved outcomes (i.e., indirect effects). Mapping the sensitivity of group-randomized designs to detect indirect or mediation effects is of critical importance because such sensitivity directly governs the types of evidence these designs can bring to bear on theories of action under realistic sample sizes (Schochet, 2011). However, from a prospective design standpoint, little is known about how to estimate the power to detect mediation in multilevel designs and the extent to which it is reasonable or even plausible to adequately power tests of mediation under different sample sizes (e.g., Vander-Weele, 2015).…”