A Review of Multisite Replication Projects in Social Psychology: Is It Viable to Sustain Any Confidence in Social Psychology’s Knowledge Base?

Abstract

Multisite (multilab/many-lab) replications have emerged as a popular way of verifying prior research findings, but their record in social psychology has prompted distrust of the field and a sense of crisis. We review all 36 multisite social-psychology replications (plus three articles reporting multiple ministudies). We start by assuming that both the original and the multisite replications were conducted in honest and diligent fashion, despite often yielding different conclusions. Four of the 36 (11%) were clearly successful in terms of providing significant support for the original hypothesis, and five others (14%) had mixed results. The remaining 27 (75%) were failures. Multiple explanations for the generally poor record of replications are considered, including the possibility that the original hypothesis was wrong; operational failure; low engagement of participants; and bias toward failure. The relevant evidence is assessed as well. There was evidence for each of the possibilities listed above, with low engagement emerging as a widespread problem (reflected in high rates of discarded data and weak manipulation checks). The few procedures with actual interpersonal interaction fared much better than others. We discuss implications in relation to manipulation checks, effect sizes, and impact on the field and offer recommendations for improving future multisite projects.

Multi-Site Replications in Social Psychology: Reflections, Implications, and Future Directions

“…Our purpose in this article is to elaborate on the results and conclusions from that earlier paper (Baumeister et al, in press). We refer interested readers to that article for full details.…”

“…An exhaustive literature search yielded 36 articles that reported a multi-site replication project in social psychology (Baumeister et al, in press). There were some additional papers filled with what we called mini-studies.…”

“…We noted that there were two big and surprising findings from the review by Baumeister et al (in press). The second one involves live interaction and is thus relevant to the generational shift in social psychology’s methods.…”

“…Concern over the frequent failures of multi-site replications prompted us to conduct a comprehensive review of all such studies in social psychology (Baumeister et al, in press). Our goal was to read the reports and seek some insight into why so many social psychology findings fail in multi-site replication.…”

Three Pathways Into Chronic Lack of Energy as a Mental Health Complaint

“…Regarding the validity of the laboratory work, it seems clear that many scientific fields suffer from problems of replicability, and social psychology’s may be especially problematic. Open Science Collaboration (2015) found social psychology experiments replicated at lower rates than cognitive psychology, and a recent survey of all 36 multi-site replications in social psychology found that only four produced significant findings consistent with the original (and most of those with reduced effect sizes; Baumeister et al, in press). Within that turbulent context, however, ego depletion has a strong case to be considered the best-replicated finding in social psychology (Baumeister & Tice, 2022).…”