The process of replication target selection in psychology: what to consider?

Abstract

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Exploring a Formal Approach to Selecting Studies for Replication: a Feasibility Study in Social Neuroscience

“…easier if researchers adhered to reporting standards, and when the relationship between statistical tests and scientific claims were more clearly specified in the article (Appelbaum et al, 2018;Lakens & DeBruine, 2021). Furthermore, it would be beneficial if researchers who perform replication studies are explicit about the reasons they selected a target study (Pittelkow et al, 2023). By exploring and documenting the wealth of information relevant to replication study selection, we can increase the ability of researchers to make well-informed decisions about which original research would be the most important to replicate.…”

“…The boundary conditions identified so far seem general enough to incorporate in the in-depth review process of replication targets by default. Future research should give us a better understanding of which additional factors to consider during in-depth review of replication candidates (e.g., Pittelkow et al, 2023).…”

Concerns about Replicability, Theorizing, Applicability, Generalizability, and Methodology across Two Crises in Social Psychology

“…The fact that almost identical concerns about the state of psychological theorizing are raised in the first and the second crisis is a source of concern, especially given the lack of practical reform that aims to improve good theorizing. Where the replication crisis has led to notable changes in the number of replication studies that are performed and published, journal policies about publishing close replication studies, funding for replication research, reflections on which studies are worthwhile to replicate (Isager et al, 2021;Pittelkow et al, 2023), transparent reporting of statistical analyses, and discussions about how to resolve ambiguities about diverging results across close replication studies (Uygun Tunç & Tunç, 2022), there are as of yet no similar concrete improvements implemented that can be expected to improve theorizing in psychology. Concrete suggestions to improve theory development in psychology deserve much greater attention.…”

Good theory from bad ingredients?

“…Journals such as the Journal of Trial and Error and the Journal in Support of the Null Hypothesis have been established to actively encourage reporting mistakes, flaws, and null findings. The research literature has become peppered with replication studies (individual as well as wide-scale) and other metaresearch studies that deal with the topic of replication (e.g., selecting which studies are most in need of replication : Field, Hoekstra, Bringmann, & van Ravenzwaaij, 2019;Pittelkow, Hoekstra, Karsten, & van Ravenzwaaij, 2021;Pittelkow et al, 2023).…”