Better Understanding the Population Size and Stigmatization of Psychologists Using Questionable Research Practices

Abstract

There has been low confidence in the replicability and reproducibility of published psychological findings. Previous work has demonstrated that a population of psychologists exists that have used questionable research practices (QRPs), or behaviors during data collection, analysis, and publication that can increase the number of false-positive findings in the scientific literature. Across two survey studies, we sought to estimate the current size of the QRP-using population of American psychologists and to identify if this sub-population of scientists is stigmatized. Using a self-report direct estimator, we estimate approximately 18\% of American psychologists have used at least one QRP in the past 12 months. We then demonstrate the use of two additional estimators: the unmatched count estimate (an indirect self-report estimator) and the generalized network scale up method (an indirect social network estimator). Additionally, attitudes of psychologists towards QRP users, and ego network data collected from self-reported QRP users, suggest that QRP users are a stigmatized sub-population of psychologists. Together, these findings provide insight into how many psychologists are using questionable practices and how they exist in the social environment.

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research.

“…While surveys remain the most feasible approach to large-scale, comprehensive assessments of perception, use, and prevalence of QRPs, asking people to answer sensitive questions about their behaviours involve several threats to validity. The available studies demonstrate that examining the perception of QRPs, let alone estimating their prevalence, is challenging and have led to contested findings [44]. In this section, we review the key methods for measuring use and prevalence of QRPs; attempts to address issues of measurement validity including social desirability; and, finally, findings on the potential causes of self-reported use of QRPs.…”

Changes in methodological study characteristics in psychology between 2010-2021

“…It should be noted, that the survey did not ask how often these practices were used, only whether they were ever used at all. Ten years later, Fox et al estimated that 18% of American psychologists have used at least one QRP in the past 12 month and that QRP users are a stigmatized sub-population of psychologists [ 36 ].…”

Quantifying, and correcting for, the impact of questionable research practices on false discovery rates in psychological science

“…The three specific practices discussed here come from a broader class that are known to be incorrect, but are nevertheless commonplace in psychology research [7][8][9][10][11]. Each is an example of a post-hoc decision being placed between data collection and publication.…”