A Brief Guide to Evaluate Replications

Abstract


 
 
 The importance of replication is becoming increasingly appreciated, however, considerably less consensus exists about how to evaluate the design and results of replications. We make concrete recommendations on how to evaluate replications with more nuance than what is typically done currently in the literature. We highlight six study characteristics that are crucial for evaluating replications: replication method similarity, replication differences, investigator independence, method/data transparency, analytic result reproducibility, and auxiliary hypotheses’ plausibility evidence. We also recommend a more nuanced approach to statistically interpret replication results at the individual-study and meta-analytic levels, and propose clearer language to communicate replication results.
 
 

Self-interest Is Overestimated: Two Successful Pre-registered Replications and Extensions of Miller and Ratner (1998)

“…The lower df for MTurk is because of randomization to the extension. The interpretation is based on LeBel et al (2019), and the size of the effects could not be precisely compared to the original due to missing information.…”

“…We compared the replication effects using established criteria (LeBel et al, 2019) and classed both samples as very close replications (Table 3).…”

“…Miller & Ratner (1998) did not report effect sizes, and they could not be calculated precisely because standard deviations or other variance measures were not reported. However, the replication effects were compared with the original (LeBel et al, 2019). There are normally three components to the interpretation.…”

Assessing and improving robustness of psychological research findings in four steps.

“…A lot has been written about ways to conduct (or evaluate) a good direct replication study (e.g., Brandt et al, 2014;LeBel et al, 2019), and in this chapter I mainly want to focus on the lessdiscussed statistical reproducibility checks. However, there are some general guidelines when doing a direct replication study that one can take into account.…”

“…Science Collaboration, 2015;Simonsohn, 2015;Verhagen & Wagenmakers, 2014;Zwaan et al, 2017). Note that several recommendations to improve the quality of replications also hold for original studies, such as transparent reporting, high statistical power, robust statistical methods, and sharing data and materials (Benjamin et al, 2018;Brandt et al, 2014;Lakens et al, 2018;Lakens & Evers, 2014;LeBel et al, 2019;Munafò et al, 2017;Nosek et al, 2012;Simmons et al, 2011).…”