Opinions on the value of direct replication: A survey of 2,000 psychologists

Abstract

A major debatehas emerged in Psychology on the merits of direct replication, with much discussion but little empirical data. Here we conducted a survey among 2,056 psychologists to gather opinions on several important themes in the debate such as: whether direct replications are valuable, whether obtained results can be the outcome of context or expertise, the intentions and reputational interests of the researchers involved, the severity of methodological problems, and what should be done to help move the field forward. Descriptively, the results of our survey suggest that most psychologists trust the seminal findings in the field and are sensitive to the contextual variables which may lead to a failed replication of such a finding; while also agreeing that more direct replications should be done. Also of interest was what opinions were related to valuing direct replications. The application of structural equation models, developed on a subsample then confirmed across subfields, suggests that the respondents’ perceptions of the severity of the methodological problems positively predicts support for direct replication, while respondents’ view on replicator’s intentions (e.g., replication bullying) negatively predicts support for direct replication.

Researchers’ Perceptions of the State of Social and Personality Psychology

“…In the wake of such efforts, researchers have sought to estimate rates of adoption of certain practices in the published literature. For example, several surveys have sought to capture researchers' self-reported attitudes towards reforms (Fuchs et al, 2012), such as conducting replication studies (Agnoli et al, 2021;Buttliere & Wicherts, 2018), and researchers' willingness to engage in particular research practices, ranging from questionable research practices (Agnoli et al, 2017;Fiedler & Schwarz, 2016;John et al, 2012) to transparency-related practices (Christensen et al, 2022;Toribio-Flórez et al, 2021;Washburn et al, 2018).…”

Survey on the Attitudes Towards and Perception of Reproducibility and Replicability in Sports and Exercise Science

“…As per the preregistration, we aimed to have a final sample size close to 2000. This sample size was based on similar surveys to our topic (Baker and Penny, 2016;Ross-Hellauer, Deppe and Schmidt, 2017;Buttliere and Wicherts, 2018). All participants were informed through the survey website that it was anonymous and voluntary.…”

“…Of 1576 researchers, 52% believed there was a significant reproducibility crisis and 38% believed there was a slight crisis in science. A similar survey of psychologists was conducted to understand the community's opinion on the importance of replication (Buttliere and Wicherts, 2018); results showed the community viewed replications as an essential aspect of the research process to determine what effects are "real". Although replication is one of the "most obvious ingredients of science" (Schmidt, 2009, p. 91), it is not the norm across all scientific disciplines causing a period of unrest amongst those who advocate for it.…”