Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions That Need to Be Powered to Detect Small Impacts

Deke, John; Wei, Thomas Y. C.; Kautz, Tim

doi:10.1080/19345747.2020.1849480

Cited by 4 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Differential attrition between experimental conditions can bias the results of experiments that use repeated measures (Deke et al., 2017 ). We therefore did a preliminary test for differential attrition prior to proceeding with longitudinal analyses on stress appraisals and predisposition for challenge seeking.…”

Section: Resultsmentioning

confidence: 99%

When do the effects of single‐session interventions persist? Testing the mindset + supportive context hypothesis in a longitudinal randomized trial

et al. 2023

View full text Add to dashboard Cite

BackgroundSingle‐session interventions have the potential to address young people's mental health needs at scale, but their effects are heterogeneous. We tested whether the mindset + supportive context hypothesis could help explain when intervention effects persist or fade over time. The hypothesis posits that interventions are more effective in environments that support the intervention message. We tested this hypothesis using the synergistic mindsets intervention, a preventative treatment for stress‐related mental health symptoms that helps students appraise stress as a potential asset in the classroom (e.g., increasing oxygenated blood flow) rather than debilitating. In an introductory college course, we examined whether intervention‐consistent messages from instructors sustained changes in appraisals over time, as well as impacts on students' predisposition to try demanding academic tasks that could enhance learning.MethodsWe randomly assigned 1675 students in the course to receive the synergistic mindsets intervention (or a control activity) at the beginning of the semester, and subsequently, to receive intervention‐supportive messages from their instructor (or neutral messages) four times throughout the term. We collected weekly measures of students' appraisals of stress in the course and their predisposition to take on academic challenges. Trial‐registration: OSF.io; DOI: 10.17605/osf.io/fchyn.ResultsA conservative Bayesian analysis indicated that receiving both the intervention and supportive messages led to the greatest increases in positive stress appraisals (0.35 SD; 1.00 posterior probability) and challenge‐seeking predisposition (2.33 percentage points; 0.94 posterior probability), averaged over the course of the semester. In addition, intervention effects grew larger throughout the semester when complemented by supportive instructor messages, whereas without these messages, intervention effects shrank somewhat over time.ConclusionsThis study shows, for the first time, that supportive cues in local contexts can be the difference in whether a single‐session intervention's effects fade over time or persist and even amplify.

show abstract

Section: Resultsmentioning

confidence: 99%

When do the effects of single‐session interventions persist? Testing the mindset + supportive context hypothesis in a longitudinal randomized trial

et al. 2023

View full text Add to dashboard Cite

show abstract

“…However, researchers forget/ignore that Cohen referred to an effect size of .2 as "difficult to detect." Even worse, education researchers such as Borman, et al (2016) are now seeking to reduce that cutoff to .1; i.e., half of difficult to detect, and some (Deke, et al 2017) are even advocating for effect size as low as .03. Perhaps the silliest rationalization for the importance of a small effect size in the education research literature is the argument that an effect size of .18 is important because it is twice as good as other interventions tested which only produced an effect size of .09.…”

Section: Perspective/theoretical Frameworkmentioning

confidence: 99%

Conditions Under Which Improvement Science/Networked Improvement Communities Produce Large Scalable Gains

Pogrow¹

2019

Proceedings of the 2019 AERA Annual Meeting

View full text Add to dashboard Cite

“…Strangely, Type I errors are inflated even with the linear functional form, to begin with, and they get worse with increased complexity in NLPE (results will be provided upon request). Finally, Deke, Wei, and Kautz (2017) brought our attention to small bias problems due to functional form misspecification in RDDs when researchers set out to devise studies that are capable of detecting a much smaller meaningful effect. Therefore, while the correct functional form is of crucial importance for devising studies sensitive to smaller effects, we may rarely need beyond quadratic functional form.…”

Section: Introductionmentioning

confidence: 99%

Minimum Detectable Effect Size Computations for Cluster-Level Regression Discontinuity Studies: Specifications Beyond the Linear Functional Form

Buluş

2021

Journal of Research on Educational Effectiveness

View full text Add to dashboard Cite

Although provides an ex-post data-driven framework for power computations in line with rdrobust Stata and R commands, which allows higher-order functional form of the score variable in non-parametric local polynomial estimation, ex-ante definitions for power computations are less clear and conventional definition of optimal design is not straightforward in cluster-level regression discontinuity (CRD) studies. This study extends power formulas proposed by Schochet ( 2008) assuming that the cluster-level score variable follows quadratic functional form. Results reveal that we need not be concerned with treatment by linear term interaction, and polynomial degree up to second order for symmetric truncation intervals. In comparison, every slight change in the functional form alters sample size requirements for asymmetric truncation intervals. Finally, an empirical framework beyond quadratic functional form is provided when the asymptotic variance of the treatment effect is untraceable. In this case, the CRD design effect is either computed from moments of the sample or approximate population moments via simulation. Formulas for quadratic functional form and the extended empirical framework are implemented in the cosa R package and companion Shiny web application.

show abstract

Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions That Need to Be Powered to Detect Small Impacts

Cited by 4 publications

References 32 publications

When do the effects of single‐session interventions persist? Testing the mindset + supportive context hypothesis in a longitudinal randomized trial

When do the effects of single‐session interventions persist? Testing the mindset + supportive context hypothesis in a longitudinal randomized trial

Conditions Under Which Improvement Science/Networked Improvement Communities Produce Large Scalable Gains

Minimum Detectable Effect Size Computations for Cluster-Level Regression Discontinuity Studies: Specifications Beyond the Linear Functional Form

Contact Info

Product

Resources

About