Selection bias, vote counting, and money-priming effects: A comment on Rohrer, Pashler, and Harris (2015) and Vohs (2015).

Abstract

When a series of studies fails to replicate a well-documented effect, researchers might be tempted to use a “vote counting” approach to decide whether the effect is reliable—that is, simply comparing the number of successful and unsuccessful replications. Vohs’s (2015) response to the absence of money priming effects reported by Rohrer, Pashler, and Harris (2015) provides an example of this approach. Unfortunately, vote counting is a poor strategy to assess the reliability of psychological findings because it neglects the impact of selection bias and questionable research practices. In the present comment, we show that a range of meta-analytic tools indicate irregularities in the money priming literature discussed by Rohrer et al. and Vohs, which all point to the conclusion that these effects are distorted by selection bias, reporting biases, or p-hacking. This could help to explain why money-priming effects have proven unreliable in a number of direct replication attempts in which biases have been minimized through preregistration or transparent reporting. Our major conclusion is that the simple proportion of significant findings is a poor guide to the reliability of research and that preregistered replications are an essential means to assess the reliability of money-priming effects.

A comprehensive meta-analysis of money priming.

“…Earlier research suggests that the money priming field contains a substantial number of underpowered studies (Vadillo, Hardwicke & Shanks, 2016). Based on four different methods, the authors argued for the presence of publication bias and p-hacking.…”

Can the Unconscious Boost Lie-Detection Accuracy?

“…But the effect has proven difficult to replicate (Rohrer, Pashler, & Harris, 2015) and there are concerns that the published effect sizes are inflated by publication or reporting biases (Vadillo, Hardwicke, & Shanks, 2016).…”

The Effect of Money Priming on Self-Focus in the Imitation-Inhibition Task

“…meta-analysis by Vadillo, Hardwicke, and Shanks (2016) suggests that the seminal findings on money priming that inspired the entire field (i.e., Vohs et al, 2006) might have been biased by selective reporting and other questionable research practices (see also Lodder, Ong, Grasman, & Wicherts, 2019). Moreover, in a recent effort to shed light onto these mixed effects, Caruso, Shapira, and Landy (2017) investigated, across three high-powered studies (N = 4,649), the degree to which different money priming manipulations affect different self-other focus related selfreports.…”