Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?

Brodeur, Abel; Cook, Nikolai; Hartley, Jonathan; Heyes, Anthony

doi:10.31222/osf.io/uxf39

Cited by 13 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To illustrate, we share some approaches that we have used in our own research, such as uploading separate preregistration-deviation lists to OSF (e.g., Atherton et al, 2022Atherton et al, , 2023, weaving deviations throughout the article (e.g., Willroth et al, 2021), creating tables that describe deviations by research question (e.g., Willroth et al, 2020;Willroth, Hill, et al, 2023), and reporting deviations in footnotes (e.g., Willroth, Luo, et al, 2023). In addition, many researchers do not report deviations at all (e.g., see Brodeur et al, 2022;Claesen et al, 2021). In fact, of 27 preregistered studies published in Psychological Science, 93% included at least one deviation from the preregistration (Claesen et al, 2021).…”

Section: The Problem and Opportunitymentioning

confidence: 99%

“…Despite most scientists' best efforts and intentions, it is sometimes necessary or preferable to deviate from preregistered plans, whether because of new innovations, data-collection errors, responding to reviewer requests, or simply correcting meaningful typos. We are not the first to document this observation or the first to make calls for preregistration deviations to be transparently reported (e.g., Brodeur et al, 2022;Campbell et al, 2023;Claesen et al, 2021;Nosek et al, , 2019Simmons et al, 2021). However, currently, there is no standardized way to report deviations from preregistrations or to evaluate the impact of those deviations on study results.…”

Section: The Problem and Opportunitymentioning

confidence: 99%

“…We found that it was difficult to identify the most transparent way to report deviations in our own work and to identify what, where, and when preregistration deviations occurred in the articles for which we served as peer reviewers. Thus, to develop a tool for standardized reporting of preregistration deviations, we drew on our own experiences with preregistration and existing unpublished templates (Brodeur et al, 2022;Campbell et al, 2023;Claesen et al, 2021;Nosek et al, , 2019Simmons et al, 2021), that led us to create a "Preregistration Deviations Table " template. On the basis of reviewer feedback on this article, we surveyed editors of psychology journals to understand current editorial perceptions and practices concerning preregistration deviations and to gather feedback on an initial draft of our template (the initial draft is available on our OSF page at https://osf.io/yhzdc).…”

Section: Developing a Tool For Standardized Reporting Of Preregistrat...mentioning

confidence: 99%

See 2 more Smart Citations

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Willroth,

Atherton

2024

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

Psychological scientists are increasingly using preregistration as a tool to increase the credibility of research findings. Many of the benefits of preregistration rest on the assumption that preregistered plans are followed perfectly. However, research suggests that this is the exception rather than the norm, and there are many reasons why researchers may deviate from their preregistered plans. Preregistration can still be a valuable tool, even in the presence of deviations, as long as those deviations are well documented and transparently reported. Unfortunately, most preregistration deviations in psychology go unreported or are reported in unsystematic ways. In the current article, we offer a solution to this problem by providing a framework for transparent and standardized reporting of preregistration deviations, which was developed by drawing on our own experiences with preregistration, existing unpublished templates, feedback from colleagues and reviewers, and the results of a survey of 34 psychology-journal editors. This framework provides a clear template for what to do when things do not go as planned. We conclude by encouraging researchers to adopt this framework in their own preregistered research and by suggesting that journals implement structural policies around the transparent reporting of preregistration deviations.

show abstract

Section: The Problem and Opportunitymentioning

confidence: 99%

Section: The Problem and Opportunitymentioning

confidence: 99%

Section: Developing a Tool For Standardized Reporting Of Preregistrat...mentioning

confidence: 99%

See 1 more Smart Citation

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Willroth,

Atherton

2024

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

show abstract

“…Attempts have been made to reduce some of these problems by pre‐registration of studies, and pre‐publication of analysis protocols. But these are not really effective (Brodeur et al., 2022; Florez et al., 2023; Sarafoglou et al., 2023), and they only concern some of the issues of trustworthiness (such as publication bias, and misuse of statistics).…”

Section: The Need For Quality Judgementsmentioning

confidence: 99%

Judging the relative trustworthiness of research results: How to do it and why it matters

Gorard

2024

Review of Education

View full text Add to dashboard Cite

This paper describes, and lays out an argument for, the use of a procedure to help groups of reviewers to judge the quality of prior research reports. It argues why such a procedure is needed, and how other existing approaches are only relevant to some kinds of research, meaning that a review or synthesis cannot successfully combine quality judgements of different types of research. The proposed procedure is based on four main factors: the fit between the research question(s) for any study and its design(s); the size of the smallest group of cases used in the headline analyses; the amount and skewness of missing data; and the quality of the data collected. This simple procedure is now relatively widely used, and has been found to lead to widespread agreement between reviewers. It can fundamentally change the findings of a review of evidence, compared to the conclusions that would emerge from a more traditional review that did not include genuine quality rating of prior evidence. And powerfully, because it is not technical, it permits users to help judge research findings. This is important as there is a growing demand for evidence‐led approaches in areas of social science such as education, wherein summaries of evidence must be as trustworthy as possible.

show abstract

“…Despite most scientists' best efforts and intentions, it is sometimes necessary or preferable to deviate from preregistered plans, whether due to new innovations, data collection errors, responding to reviewer requests, or simply correcting meaningful typos. We are not the first to document this observation nor the first to make calls for preregistration deviations to be transparently reported (e.g., Brodeur et al, 2022;Campbell et al, 2019;Claesen et al, 2021;Simmons et al, 2021). However, currently there is no standardized way to report deviations from preregistrations or to evaluate the impact of those deviations on study results.…”

Section: The Problem and Opportunitymentioning

confidence: 99%

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Willroth,

Atherton

2023

Preprint

View full text Add to dashboard Cite

Psychological scientists are increasingly using preregistration as a tool to increase the credibility of research findings. Many of the benefits of preregistration rest on the assumption that preregistered plans are followed perfectly. However, research suggests that this is the exception rather than the norm, and there are many reasons why researchers may deviate from their preregistered plans. Preregistration can still be a valuable tool, even in the presence of deviations, as long as those deviations are well-documented and transparently reported. Unfortunately, most preregistration deviations in psychology go unreported or are reported in unsystematic ways. The current paper offers a solution to this problem by providing a framework for transparent and standardized reporting of preregistration deviations, which was developed by drawing upon our own experiences with preregistration, existing unpublished templates, feedback from colleagues and reviewers, and the results of a survey of 34 psychology journal editors. This framework provides a clear template for what to do when things do not go as planned. We conclude by encouraging researchers to adopt this framework in their own preregistered research, and for journals to implement structural policies around the transparent reporting of preregistration deviations.

show abstract

Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?

Cited by 13 publications

References 0 publications

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Judging the relative trustworthiness of research results: How to do it and why it matters

Best Laid Plans: A Guide to Reporting Preregistration Deviations

Contact Info

Product

Resources

About