Author contributions: The 1 st through 4 th and last authors developed the research questions, oversaw the project, and contributed equally. The 1 st through 3 rd authors oversaw the Main Studies and Replication Studies, and the 4 th , 6 th , 7 th , and 8 th authors oversaw the Forecasting Study. The 1 st , 4 th , 5 th , 8 th , and 9 th authors conducted the primary analyses. The 10 th through 15 th authors conducted the Bayesian analyses. The first and 16 th authors conducted the multivariate meta-analysis.

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

People make inferences about others depending on the way they arrive at their moral decisions. Here, we examine evaluations of people who make moral decisions through deliberation compared to those who decide based on empathy. To do so, we turn to charitable donations. People often fail to prioritize the cost-effectiveness of charities when donating . We argue that this pattern exists in part because donors who make charitable decisions by deliberating about the cost-effectiveness of charities are perceived as less moral and less desirable as social partners than those who decide based on empathizing with the recipients of the donation. Across six pre-registered studies using two different scenarios (N = 1,961), we presented participants with descriptions of people who thought about donation decisions by either deliberating about the costeffectiveness of charities, or by deciding based on empathy. Reliably, participants judged "deliberators" to have less positive moral character and to be less desirable as social partners than "empathizers." We found these results across different designs (between-subjects and within-subjects), when evaluating respondents of different genders (male and female), and for donations of different stakes (low and high). The negative reputational effects of deliberating were reduced if "deliberators" expressed empathy first. These results suggest that there may be disincentives for selecting charities based on their impact, since people are not socially rewarded for prioritizing charitable impact but are rewarded for signaling the right kinds of moral traits. We end by discussing implications and limitations of these findings.

As part of the Many Labs 5 project, we ran a replication of van Dijk, van Kleef, Steinel, and van Beest’s (2008) study examining the effect of emotions in negotiations. They reported that when the consequences of rejection were low, subjects offered fewer chips to angry bargaining partners than to happy partners. We ran this replication under three protocols: the protocol used in the Reproducibility Project: Psychology, a revised protocol, and an online protocol. The effect averaged one ninth the size of the originally reported effect and was significant only for the revised protocol. However, the difference between the original and revised protocols was not significant.

Replication efforts in psychological science sometimes fail to replicate prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the replication protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p &lt; .05). Commenters on RP:P suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised replication protocols in multiple laboratories (Median number of laboratories per original study = XX; Range XX to YY; Median total sample = XX; Range XX to YY) for high-powered tests of each original finding with both protocols. Overall, XX of 10 RP:P protocols and XX of 10 Revised protocols showed significant evidence in the same direction as the original finding (p &lt; .05), compared to an expected XX. The median effect size was [larger/smaller/similar] for Revised protocols (ES = .XX) compared to RP:P protocols (ES = .XX), and [larger/smaller/similar] compared to the original studies (ES = .XX) and [larger/smaller/similar] compared to the original RP:P replications (ES = .XX). Overall, Revised protocols produced [much larger/somewhat larger/similar] effect sizes compared to RP:P protocols (ES = .XX). We also elicited peer beliefs about the replications through prediction markets and surveys of a group of researchers in psychology. The peer researchers predicted that the Revised protocols would [decrease/not affect/increase] the replication rate, [consistent with/not consistent with] the observed replication results. The results suggest that the lack of replicability of these findings observed in RP:P was [partly/completely/not] due to discrepancies in the RP:P protocols that could be resolved with expert peer review.