Practitioners’, Researchers’ and Students’ Trust in Research: Sensitivity to P-hacking

Abstract

The replication crisis has shown that research in psychology and other fields including biology is not as robust as previously thought. In response, methods have been introduced to address the problem and increase reproducibility, including two methods that are the focus here: (1) preregistration of study hypotheses and methods, and (2) analysis of whether p-hacking may have occurred through patterns of p-values. Each is easy to find, even in short summaries of research, but do consumers of research recognize these indicators as evidence of trustworthiness? In the current study, we examined how professionals (n = 111), researchers (n = 74) and undergraduate students (n = 78) judged the trustworthiness of short descriptions of research in their field, which varied in terms of whether there was a reference to a preregistration or evidence of potential p-hacking. Overall, participants trusted studies less when they were not preregistered. Researchers and professionals, but not students were sensitive to evidence of p-hacking. We suggest that education about questionable research practices like p-hacking and hypothesizing after the results are known needs to be improved.