Evaluation of publication bias in response interruption and redirection: A meta‐analysis

Abstract

Publication bias is the disproportionate representation of studies with large effects and statistically significant findings in the published research literature. If publication bias occurs in singlecase research design studies on applied behavior-analytic (ABA) interventions, it can result in inflated estimates of ABA intervention effects. We conducted an empirical evaluation of publication bias on an evidence-based ABA intervention for children diagnosed with autism spectrum disorder, response interruption and redirection (RIRD). We determined effect size estimates for published and unpublished studies using 3 metrics, percentage of nonoverlapping data (PND), Hedges' g, and log response ratios (LRR). Omnibus effect size estimates across all 3 metrics were positive, supporting that RIRD is an effective treatment for reducing problem behavior maintained by nonsocial consequences. We observed larger PND for published compared to unpublished studies, small and nonsignificant differences in LRR for published compared to unpublished studies, and significant differences in Hedges' g for published compared to unpublished studies, with published studies showing slightly larger effect. We found little, if any, difference in methodological quality between published and unpublished studies. While RIRD appears to be an effective intervention for challenging behavior maintained by nonsocial consequences, our results reflect some degree of publication bias present in the RIRD research literature.

Machine learning to analyze single‐case graphs: A comparison to visual inspection

“…We would advise against using graphs published in peer-reviewed journals because such datasets may result in an insufficient number of graphs showing no effect (Dowdy et al, 2020;Sham & Smith, 2014). Having a large number of graphs is typically better because it reduces the size of the confidence intervals of the results.…”

Blending as a treatment for feeding disorders: A review of the literature

“…The lack of applied research concerning this issue is unsurprising given the danger involved in treating children with feeding disorders. Moreover, research in which blending dramatically failed would likely prove difficult to publish given the preference of many journals for positive findings (i.e., publication bias; Dowdy, Tincani, & Schneider, 2020). Recent reassessment of the importance of studies with null or negative findings (Kratochwill et al, 2018) has resulted in the publication of feeding studies otherwise scuttled by ineffective treatment (Silbaugh & Swinnea, 2019).…”

Meta‐analyses and effect sizes in applied behavior analysis: A review and discussion

“…Examples of appropriate data collection methods used to calculate LRR include count, percentages, rates, and continuous duration recording. Dowdy et al (2020) calculated LRR as one of the three effect size indices included in their meta-analysis on response-interruption and redirection. Additionally, Common et al (2019) evaluated the methodological rigor and meta-analyzed research on high-probability request sequence using LRR and concluded it was potentially evidence-based.…”

“…Although additional research is needed, several promising SCED meta-analytic techniques have been proposed for detecting and assessing publication bias. One approach similar to that of Sham and Smith (2014) and Dowdy et al (2020) is to synthesize methodologically rigorous published and unpublished studies prior to calculating and comparing separate omnibus effect sizes for both categories. If the omnibus effect size and confidence interval, based on the standard error of the gray literature is less than the omnibus effect size and confidence interval of published literature, then publication bias may be evident.…”