An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis

Abstract

For over a decade, the failure to reproduce findings in several disciplines, including the biomedical, behavioral, and social sciences, have led some authors to claim that there is a so-called "replication (or reproducibility) crisis" in those disciplines. The current article examines: (a) various aspects of the reproducibility of scientific studies, including definitions of reproducibility; (b) published concerns about reproducibility in the scientific literature and public press; (c) variables involved in assessing the success of attempts to reproduce a study; (d) suggested factors responsible for reproducibility failures; (e) types of validity of experimental studies and threats to validity as they relate to reproducibility; and (f) evidence for threats to reproducibility in the behavior science/analysis literature. Suggestions for improving the reproducibility of studies in behavior science and analysis are described throughout. Keywords Reproducibility . Replication . Null hypothesis significance testing . Statistical power . Effect size measures . Statistical conclusion validity . Construct validityThe crucial role of replication is established in science generally. The undetected equipment failure, the rare and possibly random human errors of procedure, observation, recording, computation, or report are known well enough to make scientists wary of the unreplicated experiment. When we add to the possibility of the random "fluke," common to all sciences, the fact of individual organismic differences and the possibility of systematic experimenter effects in at least the

The sign effect, systematic devaluations and zero discounting

“…Replicability is essential for scientific progress (Killeen, ; Laraway et al, ). The crisis concerning failure to replicate the results from previous studies has evoked renewed attention to the issue (Ioannidis, ; Laraway et al, ). The failure to replicate has largely been due to misinterpretations or unreasonable emphasis put on statistical significance testing ( p value < .05) as a sole guide to replicability.…”

“…The failure to replicate has largely been due to misinterpretations or unreasonable emphasis put on statistical significance testing ( p value < .05) as a sole guide to replicability. Visual inspection of the original data has been advocated as an alternative approach by behavior analysts (Laraway et al, ), however with other limitations related to confirmation bias or subjectivity (Hales et al, ; Laraway et al, ). The current study discloses graphically the original data, in an attempt to be transparent and to enable both visual inspection and statistical analysis.…”

Differential reinforcement‐of‐low‐rate procedures: A systematic replication with students with autism spectrum disorder

“…Further, because DRL schedules are typically signaled in application and Becraft et al (2018) found that signals were necessary in maintaining responding with both types of DRLs, we only evaluated signaled DRL schedules. Replications of this sort are crucial in behavioral sciences given the socalled replication crisis and will ultimately improve the application of our science (Laraway, Snycerski, Pradhan, & Huitema, 2019).…”

JEAB: Past, present, and future

“…In that issue Kyonka et al (2019) updated and extended the Zimmerman et al survey and found that the linear increase of inferential statistics in JEAB articles continued through 2017, but they also noted that JEAB articles were characterized by a high frequency of graphic presentations, often with multiple panels, and most showing individual subject data. This is of critical importance because, as several authors have noted, the current "replicability crisis" in psychological and biomedical science is due, at least in part, to an over-reliance on null-hypothesis testing and thus, the "crisis" has largely been avoided in behavior analysis (Branch, 2019;Laraway, Snycerski, Pradhan & Huitema, 2019;Perone, 2019). Statistical inference is generally used in JEAB as an accompaniment to the visual presentation of primary data along with other forms of analysis.…”

“…Rejection of a null hypothesis is typically but one feature of a rich body of evidence that develops the argument for the existence of an effect. This is of critical importance because, as several authors have noted, the current "replicability crisis" in psychological and biomedical science is due, at least in part, to an over-reliance on null-hypothesis testing and thus, the "crisis" has largely been avoided in behavior analysis (Branch, 2019;Laraway, Snycerski, Pradhan & Huitema, 2019;Perone, 2019).…”