Considerations for Designing Group Randomized Trials of Professional Development With Teacher Knowledge Outcomes

Abstract

Despite recent shifts in research emphasizing the value of carefully designed experiments, the number of studies of teacher professional development with rigorous designs has lagged behind its student outcome counterparts. We outline a framework for the design of group randomized trials (GRTs) with teachers' knowledge as the outcome and consider mathematics and reading knowledge outcomes designed to assess the types of content problems that teachers encounter in practice. To estimate design parameters, we draw on a national sample of teachers for mathematics and a state Reading First sample to estimate for reading. Our results suggest that there is substantial clustering of teachers' knowledge within schools and professional development GRTs will likely need increased sample sizes to account for this clustering.

The Impact of High School Life Science Teachers’ Subject Matter Knowledge and Knowledge of Student Misconceptions on Students’ Learning

“…Unfortunately, we could not answer this question in this study (another limitation of the study). Yet it is well documented that improving teachers' SMK and PCK has been a common goal and feature among teacher professional development programs (Garet et al, 2001;Bell et al, 2010;Goldschmidt and Phelps, 2010;Kelcey and Phelps, 2013;Bausmith and Barry, 2011;Koellner and Jacobs, 2015;Lipowsky and Rzejak, 2015;Polly et al, 2015). For example, the elementary mathematics teacher preparation program at Michigan State University has multiple courses (teaching lab and field instruction) that target the development of teachers' SMK and KOSM (in the program's own words, the teacher knowledge of "learners' prior knowledge," p. 13), particularly as related to lesson planning and student assessment.…”

Statistical Power for Causally Defined Indirect Effects in Group-Randomized Trials With Individual-Level Mediators

“…In recent decades, there has been a substantial body of research detailing the estimation of statistical power to detect the main or total treatment effect across a range of experimental designs as well as practical strategies to improve the efficiency of these designs (e.g., Kelcey & Phelps, 2013; Raudenbush, 1997; Raudenbush, Martinez, & Spybrook, 2007). In part, these developments have led to a notable shift throughout the social sciences toward (group-based) experimental designs to facilitate more rigorous causal inferences.…”

“…The theory of action suggests that participation to the program improves student motivation and those improvements impact student achievement both through individual student motivation improvements and the improvement of the collective student motivation at a school. In planning this study, let us assume that we intend to sample 50 students per school ( n 1 ), expect intraclass correlation coefficients for the outcome and mediator of 0.10 (ρ), and anticipate that the covariates collected will explain about 10% of the variance in the mediator at both levels and 25% of the variance in the outcome at both levels (e.g., Hedges & Hedberg, 2007; Kelcey & Phelps, 2013). Let us hypothesize the following parameter values: The projected total treatment effect ( c ) is 0.40, the difference in mediator values between treatment and control conditions is approximately 0.5 standard deviations ( a ), the total ( B ) mediator–outcome standardized association is 0.40 with 0.10 owing to the individual-level association ( b 1 ), and assume the total treatment–mediator interaction (Δ/2) adds/subtracts 0.05 for the treatment/control group.…”

Optimal Design of Cluster- and Multisite-Randomized Studies Using Fallible Outcome Measures

“…Efficiency is an important consideration for CRTs due to the reduction in power caused by sampling clusters rather than individuals (Raudenbush, 1997). Larger samples of clusters typically yield the greatest gains in power but are also associated with substantial costs (Kelcey & Phelps, 2013a, 2013b; Raudenbush, 1997).…”