The state of social and personality science: Rotten to the core, not so bad, getting better, or getting worse?

Abstract

The scientific quality of social and personality psychology has been debated at great length in recent years. Despite research on the prevalence of Questionable Research Practices (QRPs) and the replicability of particular findings, the impact of the current discussion on research practices is unknown. The current studies examine whether and how practices have changed, if at all, over the last 10 years. In Study 1, we surveyed 1,166 social and personality psychologists about how the current debate has affected their perceptions of their own and the field's research practices. In Study 2, we coded the research practices and critical test statistics from social and personality psychology articles published in 2003-2004 and 2013-2014. Together, these studies suggest that (a) perceptions of the current state of the field are more pessimistic than optimistic; (b) the discussion has increased researchers' intentions to avoid QRPs and adopt proposed best practices, (c) the estimated replicability of research published in 2003-2004 may not be as bad as many feared, and (d) research published in 2013-2014 shows some improvement over research published in 2003-2004, a result that suggests the field is evolving in a positive direction. (PsycINFO Database Record

The case for and causes of intraminority solidarity in support for reparations: Evidence from community and student samples in Canada

“…This effect is likely to replicate given the large, national sample-a significant strength of this study is its external validity. At a time when there is widespread concern about the reliability of social psychological (and other) research (Motyl et al, 2017;Open Science Collaboration, 2015), studies such as this should bolster confidence as the large and national nature of the sample increases the chances that the results will replicate.…”

When Does HARKing Hurt? Identifying When Different Types of Undisclosed Post Hoc Hypothesizing Harm Scientific Progress

“…Certainly, it is difficult to draw strong conclusions about the replicability of an effect based on a single failed replication attempt (e.g., Maxwell, Lau, & Howard, 2015). Nonetheless, several researchers have concluded that the use of questionable research practices (John et al, 2012) has caused a unexpectedly low rate of replication in psychology and other fields (e.g., Baker, 2016;Motyl et al, 2017;Munafò et al, 2017;Świątkowski & Dompnier, 2017). The increased number of citations to Kerr's (1998) article after 2011 suggests that researchers perceive a connection between HARKing and the replication crisis.…”

“…First, it might be argued that the research literature contains an abundance of unfalsified hypotheses that may be retrieved via RHARKing (Ferguson & Heene, 2012), and that researchers often experience implicit pressures to select hypotheses that are confirmed by their results in order to increase their chances of publishing their work (e.g., Fanelli, 2010;Mazzola & Deuling, 2013;Motyl et al, 2017;Nosek et al, 2017;O'Boyle et al, 2017). Hence, RHARKing may preclude reported falsification because researchers are biased towards retrieving confirmed hypotheses rather than disconfirmed hypotheses.…”

Ideological measurement in social and political psychology

“…A major challenge in the unbiasing of our core measures of constructs such as authoritarianism, SDO, moral foundations, and disgust will be in the development of psychometrically sound and ecologically valid alternatives. In line with the ongoing debates within the field pertaining to reproducibility and bias-free theorizing (e.g., Motyl et al, 2017;Washburn et al, 2019), no formal scale drafts or sample scenario-based measures are offered in this article. However, the section that follows should be combined with the critiques presented above to formulate a potential roadmap toward the development of such tools.…”