Replication Research, Publication Bias, and Applied Behavior Analysis

Abstract

The "replication crisis" describes recent difficulties in replicating studies in various scientific fields, most notably psychology. The available evidence primarily documents replication failures for group research designs. However, we argue that contingencies of publication bias that led to the "replication crisis" also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect. The resulting research literature may unjustifiably inflate confidence about intervention effects, limit researchers' ability to delineate intervention boundary conditions, and diminish the credibility of our science. To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research.

The consecutive controlled case series: Design, data‐analytics, and reporting methods supporting the study of generality

“…In the field of genetics, massive datasets are routinely made available to reviewers and readers to enable them to confirm authors' interpretations; this is either hosted by the publishing journal or by an online data sharing platform such as Figshare (https://figshare.com). The potential value of sharing data also has been raised in the field of applied behavior analysis for the purpose of increasing transparency, promoting replication, and assuring research integrity (Hales et al, ; Tincani & Travers, ). These data sharing methods could be employed for the purpose of substantiating that SCEDs were used with every case included in a CCCS.…”

“…It is also important to document the number of cases where baseline levels are not recovered after a treatment is withdrawn during a reversal phase, as can occur as a function of carryover, learning, or the influences of other variables (although these should not be reported as failures; see Phillips et al, ). See also Tincani and Travers () for further discussion of this issue.…”

“…Given that publication bias is ubiquitous in basic and applied sciences, it is likely present in the literature on applied behavior analysis (though evidence for it is quite limited; Sham & Smith, ). The reader is referred to Tincani and Travers () and Laraway et al (2019) for further discussion of publication bias, selective data reporting, and replication research.…”

“…The reliability and generality of experimental findings have long been characterized as being of paramount importance to the analysis of behavior in basic and applied contexts (Baer et al, ; Sidman, ). Recognizing that applied research has previously focused more on ensuring reliability over generality, there is renewed interest in examining the generality of findings associated with clinical procedures (Hanley, ; Tincani & Travers, ).…”

Generative time telling in adults with disabilities: A matrix training approach

“…These levels, however, demonstrate that matrix training (as planned for in this study and with these participants) did not produce recombinative generalization for all untrained cells. Publishing these results, although modest, allow practitioners and researchers to “…examine how varied intervention procedures, dosages, contexts, and participant responses modulate intervention effects…” (Tincani & Travers, , p. 67). There is also a huge disparity of intervention research that focuses on adults with autism (Edwards, Watkins, Lotfizadeh, & Poling, ), including matrix training studies.…”