Replication value as a function of citation impact and sample size

Abstract

Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication e?ort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2020). In this article, we propose a quantitative operationalization of replication value. Wefirst discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.

Assessing the replication landscape in experimental linguistics

“…Finding convenient yet effective tools to identify worthwhile replication tar-gets is an active meta-scientific field (e.g., Coles et al, 2018;Hardwicke et al, 2018;Isager et al, 2021a) and feasible algorithms are currently developed and tested (Isager et al, 2021b). When it comes to more accessible ways to conduct replication studies, several authors have suggested involving our students more rigorously (Frank and Saxe, 2012;Grahe et al, 2012;e.g., Leeuw et al, 2019;Roettger and Baer-Henney, 2019), possibly creating a rich learning experience for our students while at the same time reducing the resource costs of replication studies.…”

Reproducible research practices and transparency across linguistics

“…For example, journals could dedicate space to direct replications (e.g., as its own article type or journal section). Researchers, on the other hand, need to establish resource-efficient ways to identify replication targets (84) and conduct them (11,85).…”

Reproducible research practices and transparency across linguistics

“…as its own article type or journal section). Researchers, on the other hand, need to establish resource-efficient ways to identify replication targets (84) and conduct them (11,85).…”