Preregistration Is Hard, And Worthwhile

Nosek, Brian A.; Beck, Emorie D; Campbell, Lorne; Flake, Jessica Kay; Hardwicke, Tom E; Mellor, David Thomas; Veer, Anna E. van ‘t; Vazire, Simine

doi:10.1016/j.tics.2019.07.009

Cited by 257 publications

(223 citation statements)

References 15 publications

Supporting

Mentioning

207

Contrasting

Unclassified

Order By: Relevance

“…Pre-registration fulfils a number of potential functions (Nosek et al, 2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al, 2014;Wagenmakers et al, 2012) and enabling the detection and mitigation of questionable research practices like selective outcome reporting (Franco et al, 2016;John et al, 2012;Simmons et al, 2011). Pre-registration is relatively new to psychology (Nosek et al, 2018;Nosek et al 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012) where they have become the expected norm (Zarin et al, 2017). However, clinical trials represent only a minority of biomedical research, and estimates suggest that pre-registration is overall rare in biomedicine (Iqbal et al, 2016;.…”

Section: Discussionmentioning

confidence: 99%

“…It is important to note that because there is a time-lag between registration and study publication, our measures may underestimate adoption. Whilst norms and standards for pre-registration in psychology are still evolving (Nosek et al, 2019), several dedicated registries, like the OSF, will host preregistrations and detailed guidance is available (Klein et al, 2018).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

Hardwicke¹,

Thibault²,

Kosie

et al. 2020

Preprint

View full text Add to dashboard Cite

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives have emphasized the benefits of several transparency and reproducibility-related research practices; however, their adoption across the psychology literature is unknown. To estimate their prevalence, we manually examined a random sample of 250 psychology articles published between 2014-2017. Over half of the articles were publicly available (154/237, 65% [95% confidence interval, 59%-71%]); however, sharing of research materials (26/183, 14% [10%-19%]), study protocols (0/188, 0% [0%-1%]), raw data (4/188, 2% [1%-4%]), and analysis scripts (1/188, 1% [0%-1%]) was rare. Pre-registration was also uncommon (5/188, 3% [1%-5%]). Many articles included a funding disclosure statement (142/228, 62% [56%-69%]), but conflict of interest statements were less common (88/228, 39% [32%-45%]). Replication studies were rare (10/188, 5% [3%-8%]) and few studies were included in systematic reviews (21/183, 11% [8%-16%]) or meta-analyses (12/183, 7% [4%-10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish a baseline which can be used to assess future progress towards increasing the credibility and utility of psychology research.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

Hardwicke¹,

Thibault²,

Kosie

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Taken together, questionable research or reporting practices can range from accidental omission of information to fraud, and include justifiable practices, ignorant and negligent practices, practices meant to mislead and misrepresent, downright lying, and many combinations. Consistent with calls for disclosure (Simmons et al, 2011), preregistration (Cybulski et al, 2016;Nosek et al, 2019), and open science more generally (Aguinis et al, 2018;LeBel et al, 2017;Nosek & Bar-Anan, 2012), transparency goes a long way toward confirming or allaying doubts about the validity of study conclusions. These calls for transparency, as well as reporting guidelines (e.g., CONSORT), prescribe the transparent reporting of measurement practices.…”

Section: Qrps Have Been Defined As the Exploitation Of Ambiguities Inmentioning

confidence: 92%

Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

Flake¹,

Fried²

2019

Preprint

138

130

View full text Add to dashboard Cite

In this paper, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

show abstract

“…We have become better at identifying and preventing questionable research practices such as p-hacking and HARKing (hypothesizing after results are known) through tools like preregistration, registered reports, and sharing of data and code (O. Klein et al, 2018;Nosek et al, 2019). Identified challenges and proposed solutions have focused on increasing the reliability and replicability of psychological findings by improving methodological and statistical practices (Muthukrishna & Henrich, 2019).…”

Section: Data Rich and Theory Poormentioning

confidence: 99%

Lack of theory building and testing impedes progress in the factor and network literature

Fried¹

2020

Preprint

113

138

View full text Add to dashboard Cite

The last decade has brought reforms to improve methodological practices, with the goal to increase the reliability and replicability of effects. However, explanations of effects remain scarce, and a growing chorus of scholars argues that the replicability crisis has distracted from a crisis of theory. In the same decade, the empirical literature using factor and network models has grown rapidly. I discuss three ways in which this literature falls short of theory building and testing. First, statistical and theoretical models are conflated, leading to invalid inferences such as the existence of psychological constructs based on factor models, or recommendations for clinical interventions based on network models. I demonstrate this inferential gap in a simulation study on statistical equivalence: excellent model fit does little to corroborate a theory, regardless of quality or quantity of data. Second, researchers fail to explicate theories about psychological constructs, but use implicit causal beliefs to guide inferences. These latent theories have led to problematic best practices in psychological research where inferences are drawn based on one specific causal model that is assumed, but not explicated. Third, explicated theories are often weak theories: narrative and imprecise descriptions vulnerable to hidden assumptions and unknowns. They fail to make clear predictions, and it remains unclear whether statistical effects corroborate such theories or not. Weak theories are immune to refutation or revision. I argue that these three challenges to theory building and testing are common and harmful, and impede theory formation, failure, and reform. A renewed focus on theoretical psychology and formal models offers a way forward.

show abstract

Preregistration Is Hard, And Worthwhile

Cited by 257 publications

References 15 publications

Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

Lack of theory building and testing impedes progress in the factor and network literature

Contact Info

Product

Resources

About