Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?

Abstract

There is increased concern about poor scientific practices arising from an excessive focus on P-values. Two particularly worrisome practices are selective reporting of significant results and ‘P-hacking’. The latter is the manipulation of data collection, usage, or analyses to obtain statistically significant outcomes. Here, we introduce the novel, to our knowledge, concepts of selective reporting of nonsignificant results and ‘reverse P-hacking’ whereby researchers ensure that tests produce a nonsignificant result. We test whether these practices occur in experiments in which researchers randomly assign subjects to treatment and control groups to minimise differences in confounding variables that might affect the focal outcome. By chance alone, 5% of tests for a group difference in confounding variables should yield a significant result (P < 0.05). If researchers less often report significant findings and/or reverse P-hack to avoid significant outcomes that undermine the ethos that experimental and control groups only differ with respect to actively manipulated variables, we expect significant results from tests for group differences to be under-represented in the literature. We surveyed the behavioural ecology literature and found significantly more nonsignificant P-values reported for tests of group differences in potentially confounding variables than the expected 95% (P = 0.005; N = 250 studies). This novel, to our knowledge, publication bias could result from selective reporting of nonsignificant results and/or from reverse P-hacking. We encourage others to test for a bias toward publishing nonsignificant results in the equivalent context in their own research discipline.

Cultural Psychology, Diversity, and Representation in Open Science

“…Identifying the cultural model is the first step to working through the next problem: identifying the mechanism that would account for any observed group differences. Most, but certainly not all, group-comparison research hypothesizes the presence rather than absence of group differences (but see Chuard et al, 2019). At the same time, most, but certainly not all, of such research does not measure and test the putative mechanism that accounts for those differences.…”

DamianSpengler_2020_EJP_Negligible Effects of Birth Order on Selection into Scientific and Artistic Careers, Creativity, and Status Attainment

“…We opted to keep the alpha level at .05 for three reasons. First, .05 was the alpha level we set a priori, and we were concerned that lowering the alpha level after seeing the results could have been a case of reverse p-hacking (Chuard et al, 2019), given that, in this case, fewer (not more) statistically significant coefficients would have bolstered our conclusion that birth order effects are negligible. Second, we presented all results from all analyses conducted, with exact p-values, so readers can assess the results for themselves.…”

The inner workings of Registered Reports

“…But pursuing phenomena should not replace theory testing, as it is only in the context of a theoretical research program that science develops (Alger, 2019;Deutsch, 2011;Guest & Martin, 2020;Lakatos, 1970). Finally, when data are seen before planning analyses, p-hacking (or B-hacking for Bayes factors) may make support for the predictions sufficiently probable regardless of the data that the data, as analysed, are no longer evidential regarding phenomena ( Figure 1D) (Chuard, Vrtílek, Head, & Jennions, 2019;Mayo, 2018;Simmons, Nelson, & Simonsohn, 2011).…”