Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Abstract

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (1) the value of being certain about the claim, and (2) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.

Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science

“…Researchers have focused attention on how replication studies should be conducted and how their results should be interpreted [9][10][11][12]. Yet, there has been little discussion about the factors influencing study selection for replication and even less of a consensus on what those factors should be [1]. Moreover, researchers are limited by energy and resource constraints; selecting studies for replication that have the most potential to teach us something useful is therefore paramount.…”

“…An important goal of science is to advance knowledge, through the generation of novel empirical claims that are robustly supported [ 1 , 2 ]. For an experimentally derived claim to be robustly supported, it should, in principle, be replicable.…”

“…Studies might be chosen for replication based on a need to reduce existing uncertainty around the statistical soundness of reported findings, based, for example, on inconsistencies in the statistical results, inflated effect sizes with wide confidence intervals, conflicting previous results, and the prevalence of questionable research practices (e.g. p-hacking) [ 1 , 23 ]. Studies could also be targeted for replication based on their theoretical or practical value, as indicated by their scholarly or public impact (e.g., Altmetric attention score or citation impact) [ 25 ].…”

“…Guidelines were proposed to determine what studies are most worthwhile to replicate based on a Bayesian decision-making framework [ 24 , 27 – 29 ] or the proposed replication value of the study (i.e., what we stand to learn by replicating the finding) [ 1 , 23 , 30 ]. Although Isager and colleagues admit no guidelines can “ provide a single set of rules for deciding what to replicate in all circumstances ” [ 1 ], there should be some justification for the selection of studies to replicate.…”

“…Guidelines were proposed to determine what studies are most worthwhile to replicate based on a Bayesian decision-making framework [ 24 , 27 – 29 ] or the proposed replication value of the study (i.e., what we stand to learn by replicating the finding) [ 1 , 23 , 30 ]. Although Isager and colleagues admit no guidelines can “ provide a single set of rules for deciding what to replicate in all circumstances ” [ 1 ], there should be some justification for the selection of studies to replicate. For a large-scale replication project such as that proposed here, the selection of studies to replicate should be based on the aims of the project, namely, to provide an initial estimate of the overall replicability of studies within the entire field.…”