Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures

Kerwin, Jason; Thornton, Rebecca

doi:10.1162/rest_a_00911

Cited by 73 publications

(45 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4 Of course, it is not easy to produce such outcomes. Kerwin and Thornton (2018) describe how producing an effective intervention requires determining the near optimal mix of design parameters, and how minor changes in any parameter can substantially reduce effectiveness. They show how an early literacy intervention in northern Uganda found major gains in reading and writing.…”

Section: Practical Benefit and Improvement Sciencementioning

confidence: 99%

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

Pogrow

2019

The American Statistician

View full text Add to dashboard Cite

Relying on effect size as a measure of practical significance is turning out to be just as misleading as using p-values to determine the effectiveness of interventions for improving clinical practice in complex organizations such as schools. This article explains how effect sizes have misdirected practice in education and other disciplines. Even when effect size is incorporated into RCT research the recommendations of whether interventions are effective are misleading and generally useless to practitioners. As a result, a new criterion of practical benefit is recommended for evaluating research findings about the effectiveness of interventions in complex organizations where benchmarks of existing performance exist. Practical benefit exists when the unadjusted performance of an experimental group provides a noticeable advantage over an existing benchmark. Some basic principles for determining practical benefit are provided. Practical benefit is more intuitive and is expected to enable leaders to make more accurate assessments as to whether published research findings are likely to produce noticeable improvements in their organizations. In addition, practical benefit is used routinely as the research criterion for the alternative scientific methodology of improvement science that has an established track record of being a more efficient way to develop new interventions that improve practice dramatically than RCT research. Finally, the problems with practical significance suggest that the research community should seek different inferential methods for research designed to improve clinical performance in complex organizations, as compared to methods for testing theories and medicines.

show abstract

Section: Practical Benefit and Improvement Sciencementioning

confidence: 99%

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

Pogrow

2019

The American Statistician

View full text Add to dashboard Cite

show abstract

“…In some cases, researchers have directly designed the evaluation to learn about the scale up effect. Kerwin and Thornton (2018) include a novel treatment arm in their evaluation of an education program in Uganda to model the policy-relevant situation. They find large effects of a high-quality, relatively high-cost program, but find that a less expensive program specifically designed to scale is much less effective.…”

Section: Examplesmentioning

confidence: 99%

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

et al. 2019

View full text Add to dashboard Cite

Policymakers are increasingly turning to insights gained from the experimental method as a means to inform large scale public policies. Critics view this increased usage as premature, pointing to the fact that many experimentally-tested programs fail to deliver their promise at scale. Under this view, the experimental approach drives too much public policy. Yet, if policymakers could be more confident that the original research findings would be delivered at scale, even the staunchest critics would carve out a larger role for experiments to inform policy. Leveraging the economic framework of Al-Ubaydli et al. (2019), we put forward 12 simple proposals, spanning researchers, policymakers, funders, and stakeholders, which together tackle the most vexing scalability threats. The framework highlights that only after we deepen our understanding of the scale up problem will we be on solid ground to argue that scientific experiments should hold a more prominent place in the policymaker's quiver.

show abstract

“…For this group, the estimated treatment impacts on P7 attendance, PLE participation, and Passing the PLE are .024, .014, and -.0007 respectively, and the p-values associated with these effects are all .6 or greater.43Kerwin and Thornton (2018) report that details concerning training and resource provision interact in complicated ways that impact the effectiveness of the Mango Tree literacy program.44 SeeGlewwe et al (2009),Glewwe and Muralidharan (2016).…”

mentioning

confidence: 98%

Educator Incentives and Educational Triage in Rural Primary Schools

et al. 2019

View full text Add to dashboard Cite

In low-income countries, primary school student achievement is often far below grade level and dropout rates remain high. Further, some educators actively encourage weaker students to drop out before reaching the end of primary school to avoid the negative attention that a school receives when its students perform poorly on their national primary leaving exams. We report the results of an experiment in rural Uganda that sought to both promote learning and reduce dropout rates. We offered bonus payments to grade six (P6) teachers that rewarded each teacher for the math performance of each of her students relative to comparable students in other schools. This Pay for Percentile (PFP) incentive scheme did not improve overall P6 math performance, but it did reduce dropout rates. PFP treatment raised attendance rates a full year after treatment ended from .56 to .60. In schools with math books, treatment increased these attendance rates from .57 to .64, and PFP also improved performance on test items covered by P6 books. PFP did not improve any measure of attendance, achievement, or attainment in schools without books.

show abstract

Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures

Cited by 73 publications

References 48 publications

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

Educator Incentives and Educational Triage in Rural Primary Schools

Contact Info

Product

Resources

About