“I know it's bad but I have been pressured into it”: Questionable research practices among psychology students in Canada

Abstract

Background: Questionable research practices (QRPs) have been identified as a driving force of the replication crisis in the field of psychological science. The aim of this study was to assess the frequency of QRP use among psychology students in Canadian universities, and to better understand reasons and motivations for QRP use.Method: Participants were psychology students attending Canadian universities and were recruited via online advertising and university email invitations to complete a bilingual survey. Respondents were asked how often they and others engaged in seven QRPs. They were also asked to estimate the proportion of psychology research impacted by each QRP and how acceptable they found each QRP. Data were collected through Likert-scale survey items and open-ended text responses between May 2020 and January 2021, and was analyzed using descriptive statistics and thematic analysis. Results: 425 psychology students completed the survey. The sample consisted of 40% undergraduate students, 59% graduate students and 1% post-doctoral fellows. Overall, 64% of participants reported using at least one QRP, while 79% reported having observed others engaging in at least one QRP. The most frequently reported QRPs were p-hacking (46%), not submitting null results for publication (31%), excluding outcome measures (30%), and hypothesizing after results are known (27%). These QRPs were also the most frequently observed in others, estimated to be the most prevalent in the field, and rated as the most acceptable. Qualitative findings show that students reported that pressures to publish motivated their QRP use, with some reporting that certain QRPs are justifiable in some cases (e.g., in the case of exploratory research). Students also reported that QRPs contribute to the replication crisis and to publication bias and offered several alternatives and solutions to engaging in QRPs, such as gaining familiarity with open science practices. Conclusions: Most Canadian psychology students in this sample report using QRPs, which is unsurprising since they observe such practices in their research environment and estimate that they are prevalent. In contrast, most students believe that QRPs are not acceptable. The results of this study highlight the need to examine the pedagogical standards and cultural norms in academia that may promote or normalize QRPs in psychological science, to improve the quality and replicability of research in this field.

Questionable Research Practices, Low Statistical Power, and Other Obstacles to Replicability: Why Preclinical Neuroscience Research Would Benefit from Registered Reports

“…• The prevalence of QRPs is difficult to know, but the evidence is not optimistic. A survey of 2155 psychologists for their estimates of the prevalence of and their own engagement in 10 QRPs showed self-admission rates ranging from 0.6% to 63.4%, with prevalence estimates 5-20% higher for all but one QRP (John et al, 2012).These results were replicated in a cohort of 277 Italian psychologists with a translated questionnaire (Agnoli et al, 2017), and largely agree with several other surveys of QRP engagement (for a summary, see Table S5 in a review by Nosek and colleagues;Nosek et al, 2021; see also Fanelli, 2009;Banks et al, 2016;Fiedler and Schwarz, 2016;Motyl et al, 2017;Krishna and Peter, 2018;Janke et al, 2019;Rabelo et al, 2020;Moran et al, 2021).…”

Quality assessment of scientific manuscripts in peer review and education

“…Over 90% of psychology researchers admitted having engaged in QRPs during their career (Agnoli et al, 2017;John, et al, 2012;Makel, et al, 2021;Rabelo et al, 2020). Estimates for students indicate that 64% of them have already engaged in QRPs in their short academic careers (up to graduate level, Moran et al, 2021), whereas 40% of psychology students engaged in QRPs in their thesis (Krishna & Peter, 2018). It is currently unknown how academics differ in their quality assessment of theses and articles, and how potential QRPs affect this assessment.…”

What do Incoming University Students Believe About Open Science Practices in Psychology?

“…On the other hand, some evidence suggests that undergraduate students may quickly begin to engage in practices that hamper reproducibility. For example, Moran et al (2021) found that 26.5% of the undergraduate students in their sample admitted to "Conducting multiple statistical analyses on the same dataset in an attempt to find a statistically significant result" (p. 14; i.e., p-hacking), while 9.6% reported rounding down pvalues to meet a significance threshold. Knowing the extent to which incoming psychology students may-or may not-be "naive open scientists" could help guide pedagogical approaches.…”