Replicability, Robustness, and Reproducibility in Psychological Science

Abstract

Replication, an important, uncommon, and misunderstood practice, is making a comeback in psychology. Achieving replicability is a necessary but not sufficient condition for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understanding to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understanding and observed surprising failures to replicate many published findings. Replication efforts also highlighted sociocultural challenges, such as disincentives to conduct replications, framing of replication as personal attack rather than healthy scientific practice, and headwinds for replication contributing to self-correction. Nevertheless, innovation in doing and understanding replication, and its cousins, reproducibility and robustness, have positioned psychology to improve research practices and accelerate progress.

Assessing and improving robustness of psychological research findings in four steps.

“…When interpreting a reproducibility failure in one of the first three steps above (checking internal inconsistencies, reanalysis, or sensitivity analyses), it matters whether one encountered a process or outcome reproducibility failure. Remember that a process reproducibility failure occurs when not all steps could be followed to redo the original analysis, whereas an outcome reproducibility failure occurs when the outcome of the reanalysis shows a different result than the one originally reported (Nosek et al, 2021). An outcome reproducibility failure is a more clear-cut outcome than a process reproducibility failure: in the former case one can conclude that a reported result is not robust, whereas in the latter case one cannot assess the robustness of a result.…”

“…This is often the goal in multi-lab replication projects, such as the Reproducibility Project: Psychology (Open Science Collaboration, 2015). In these cases, the replication studies are often high-powered and pre-registered, arguably enhancing their evidential value compared to the original study they are replicating (Nosek et al, 2021). They therefore attempt to provide a more or less definitive conclusion about the robustness of the original result.…”

“…Second, an outcome reproducibility failure occurs when the original analysis can be repeated but leads to a different result than the one that is reported (Nosek et al, 2021). In general, it is plausible to assume that reanalyzing the same data according to the same methods leads to the same results.…”

“…Roughly, there can be two reasons why a result is not statistically reproducible (Nosek et al, 2021). First, a process reproducibility failure occurs when it is not possible to repeat the steps of the original analysis, for example because of unavailable data or code, unclear description of the analysis steps, or unavailable software or tools.…”

Reducing bias, increasing transparency, and calibrating confidence with preregistration

“…Signatures of bias have also been detected in the published literature; in particular, the suspicious juxtaposition of overwhelmingly statistically significant results [181][182][183] and overwhelmingly inadequate statistical power 184,185 . The results of multiple large-scale replication studies consistently imply that many published results are either exaggerated by bias or entirely spurious 82,[186][187][188][189] .…”