The logical structure of experiments lays the foundation for a theory of reproducibility

Abstract

The scientific reform movement has proposed openness as a potential remedy to the putative reproducibility or replication crisis. However, the conceptual relationship between openness, replication experiments, and results reproducibility has been obscure. We analyze the logical structure of experiments, define the mathematical notion of idealized experiment, and use this notion to advance a theory of reproducibility. Idealized experiments clearly delineate the concepts of replication and results reproducibility, and capture key differences with precision, allowing us to study the relationship among them. We show how results reproducibility varies as a function of: the elements of an idealized experiment, the true data generating mechanism, and the closeness of the replication experiment to an original experiment. We clarify how openness of experiments is related to designing informative replication experiments and to obtaining reproducible results. With formal backing and evidence, we argue that the current "crisis" reflects inadequate attention to a theoretical understanding of results reproducibility.

Generalizations: The grail and the gremlins.

“…We also can and should be clear about which aspects of our results are likely to generalize and which aspects are not, and the reasons why. To be most fruitful, the speculations should be guided by strong theory to select the parameters to test as well as the expectations of when something should versus should not be expected to generalize (e.g., Buzbas et al, 2022; Holleman et al, 2020; Jamieson & Pexman, 2020). We also can and should suggest future research that would address the likely limitations.…”

“…Better descriptions of behavioral variability will lead to the next—arguably more challenging—step, which is to include in our research measures and tasks expected to relate to the outcome variables of primary interest, thus helping to explain the individual differences we observe (see Beck & Jackson, 2022, for successful application of a person-specific, idiographic approach to prediction of future behaviors and experiences). This requires that we have strong theory (Buzbas et al, 2022) and clear hypotheses—or at least, educated guesses—regarding potential sources of variability. And of course, establishing a potential source of variability also entails elimination of other possible sources, lest we establish that everything is related to everything.…”

Bridging the gap between formal theory and scientific reform practices

“…Once again, hand-waving at perceived closeness or likeness will introduce further obscurity and make it more difficult to properly interpret the replication results. In Buzbas et al (2022), we show how easily true or false results can be made more or less reproducible with small perturbations in study design.…”

“…When replications are non-exact, there is no single true reproducibility rate for results from all studies and therefore, no single sampling distribution of reproducibility rate. The mean of the process, then, becomes the target, but it needs to be interpreted carefully (see Buzbas et al, 2022, for more information).…”

“…For example, it should be clear that the true reproducibility rate of a true result in a sequence of exact replication studies with independent and identically distributed random samples can take any value on [0, 1] depending on the elements defining that study. Thus, we should not have a default expectation of high rates of reproducibility or assume that low reproducibility is indicative of false discoveries (Buzbas et al, 2022).…”