Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Abstract

Here we provide further details on the replications, the estimation of standardized effect sizes and complementary replicability indicators, the implementation of the prediction markets and surveys, the comparison of prediction market beliefs, survey beliefs, and replication outcomes, the comparison of reproducibility indicators to experimental economics and the psychological sciences, and additional results and data for the individual studies and markets. The code used for the estimation of replication power, standardized effect sizes, all complementary replication indicators, and all results is posted at OSF (https://osf.io/pfdyw/). Replications Inclusion criteriaWe replicated 21 experimental studies in the social sciences published between 2010 and 2015 in Nature and Science. We included all studies that fulfilled our inclusion criteria for:(i) the journal and time period, (ii) the type of experiment, (iii) the subjects included in the experiment, (iv) the equipment and materials needed to implement the experiment, and (v) the results reported in the experiment. We did not exclude studies that had already been subject to a replication, as this could affect the representativity of the included studies. We define and discuss the five inclusion criteria below. Journal and time period: We included experimental studies published in Nature andScience between 2010 and 2015. The reason for focusing on these two journals is that they are typically considered the two most prestigious general science journals. Articles published in these journals are considered exciting, innovative, and important, which is also reflected in their high impact factors. * Number of observations; number of individuals provided in parenthesis. † Replicated; significant effect (p < 0.05) in the same direction as in original study. ‡ Statistical power to detect 50% of the original effect size r. § Relative standardized effect size. * Belief about the probability of replicating in stage 1 (90% power to detect 75% of the original effect size).† Predicted added probability of replicating in stage 2 (90% power to detect 50% of the original effect size) compared to stage 1. * Mean number of tokens (points) invested per transaction. † Mean number of shares bought or sold per transaction.

The gap between medical and monetary choices under risk persists in decisions for others

“…Specifically, probability information tends to be neglected in medical decisions. Such a replication of previous findings is especially valuable in light of the recent replication crisis in the social sciences (Camerer et al, 2018;Open Science Collaboration, 2015). As a novel contribution, our results showed that the gap between medical and monetary decisions generalizes to choices for others: Choice behavior and information search in medical decisions differ from monetary decisions irrespective of who is affected by the outcome.…”

Philosophy of science and the replicability crisis

“…F is the finding that donors are more likely to donate to a charity in the first situation. Imagine we want to replicate this finding directly (as Camerer et al, 2018) did. Changing the donation amount might make a difference; hence, the replication would not be direct, but whether we conduct the replication in a room with gray or white walls should be irrelevant.…”

“…Moreover, the Reproducibility Project (Open Science Collaboration, 2015) studied a random sample of published studies to estimate the replicability of psychology more generally. Similar projects have assessed the replicability of cancer research (Nosek & Errington, 2017), experimental economics (Camerer et al, 2016), and studies from the prominent journals Nature and Science (Camerer et al, 2018). These studies give us an unsettling perspective.…”

A New Standard for the Analysis and Design of Replication Studies

“…As a consequence of the so-called replication crisis (Ioannidis, 2005;Begley and Ioannidis, 2015), recent years have witnessed increasing interest in large-scale replication projects, e.g. Open Science Collaboration (2015) and Camerer et al (2016Camerer et al ( , 2018. Such efforts help to assess to what extent claims of new discoveries can be confirmed in independent replication studies whose procedures are as closely matched to the original studies as possible.…”