Registered trials report less beneficial treatment effects than unregistered ones: a meta-epidemiological study in orthodontics

Papageorgiou, Spyridon N.; Xavier, Guilherme M.; Cobourne, Martyn T.; Eliades, Theodore

doi:10.1016/j.jclinepi.2018.04.017

Cited by 29 publications

(16 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results showed that 57% of the studies published before 2000 showed a significant benefit of the intervention, as opposed to only 8% of trials published after 2000. Similar results were 9 reported in a (preregistered) meta-analysis of meta-analyses of orthodontics and dentofacial orthopedics studies: registered trials reported less favorable intervention effects compared to unregistered trials (Papageorgiou et al, 2018) . Preregistration can also help identify whether funding sources are correlated with study outcome, potentially uncovering questionable practices due to (undisclosed) conflicts of interest.…”

Section: Advantages Of Preregistrationsupporting

confidence: 85%

Making ERP Research More Transparent: Guidelines for Preregistration

Paul¹,

Govaart²,

Schettino³

2020

Preprint

View full text Add to dashboard Cite

Confirmation bias and pressure to publish may prompt the (unconscious) exploration of various methodological options and reporting only the ones that lead to a (statistically) significant outcome. This undisclosed analytic flexibility increases false positive results and inflates effect sizes, ultimately creating a skewed representation of knowledge. This issue is particularly relevant in EEG research, where a myriad of preprocessing and analysis pipelines can be used to extract information from complex multidimensional data. One solution to limit undisclosed analytic flexibility is preregistration: researchers write a time-stamped, publicly accessible research plan with hypotheses, data collection plan, and intended preprocessing and statistical analyses before the start of a research project. In this manuscript, we present an overview of the problems associated with undisclosed analytic flexibility (particularly in human neuroimaging and electrophysiology), discuss why and how EEG researchers would benefit from adopting preregistration, provide guidelines and examples on how to preregister data preprocessing and analysis steps in typical ERP studies, and conclude by addressing common rebuttals as well as clarifying possibilities and limitations of this open science practice.

show abstract

Section: Advantages Of Preregistrationsupporting

confidence: 85%

Making ERP Research More Transparent: Guidelines for Preregistration

Paul¹,

Govaart²,

Schettino³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…It is worth mentioning that only two of the studies under consideration in the systematic review reported being registered trials, none of which could be included in the meta-analysis. Several meta-research studies show that effect sizes tend to be substantially smaller in registered trials (Kaplan and Irvin 2015;Papageorgiou et al 2018), possibly due to selective reporting and other biases in unregistered research that artificially inflate effect sizes (Kerr 1998;Simmons et al 2011). Ideally, future RCTs conducted in this topic should adhere to preregistered protocols and analysis plans, to ensure that their results are free from these sources of bias.…”

Section: Discussionmentioning

confidence: 99%

Does Mindfulness Meditation Training Enhance Executive Control? A Systematic Review and Meta-Analysis of Randomized Controlled Trials in Adults

2019

View full text Add to dashboard Cite

Objectives: Over the last years, mindfulness meditation has been claimed to be effective in enhancing several cognitive domains, including executive control. However, these claims have been mostly based on findings pertaining to case-control and cross-sectional studies, which are by nature unable to reveal causal relationships. Aiming to address this issue, we set out to conduct the first quantitative assessment of the literature concerning mindfulness meditation as an enhancer for executive control considering only randomized controlled studies. Methods: We conducted a systematic review and meta-analysis covering experimental studies testing the effect of mindfulness meditation training on at least one executive control function (working memory, inhibitory control, or cognitive flexibility) in adult samples. Four databases were examined, resulting in the identification of 822 candidate references. After a systematic filtering process, a set of 16 studies was retained for evaluation, of which 13 could be included in a subsequent meta-analysis. Results: We found an average effect size of g = 0.34 [0.16, 0.51], indicating a small-to-medium effect of mindfulness meditation training in enhancing executive control. Effect sizes for individual functions were g = 0.42 [0.10, 0.74] for working memory, g = 0.42 [0.20, 0.63] for inhibitory control, and g = 0.09 [-0.13, 0.31] for cognitive flexibility. Funnel plot asymmetry analysis revealed no evidence of publication bias. Conclusions: Taken together, our findings provide preliminary and moderate yet positive evidence supporting the enhancing effects of mindfulness meditation on executive control. Shortcomings of included studies and considerations for future empirical and meta-analytical research are discussed.

show abstract

“…Our analysis showed that direction of primary outcomes, i.e., statistically significant or not, doesn’t seem to be related with the registration status. Previous studies evaluating association between trial registration and treatment effect size have reported inconsistent results [21,46] and another recent study reported that direction of primary outcome was not subject to the status of registration [47]. Given these inconsistent results, a larger meta-epidemiological and/or cross-sectional study encompassing various medical disciplines is needed to determine the association between trial registration and treatment effect size and/or direction of study outcomes.…”

Section: Discussionmentioning

confidence: 99%

Trial registration as a safeguard against outcome reporting bias and spin? A case study of randomized controlled trials of acupuncture

et al. 2019

View full text Add to dashboard Cite

Background and objectiveTrial registration is widely endorsed as it is considered not only to enhance transparency and quality of reporting but also to help safeguard against outcome reporting bias and probably spin, known as specific reporting that could distort the interpretation of results thus mislead readers. We planned to investigate the current registration status of recently published randomized controlled trials (RCTs) of acupuncture, outcome reporting bias in the prospectively registered trials, and the association between trial registration and presence of spin and methodological factors in acupuncture RCTs.MethodsAcupuncture RCTs published in English in recent 5 years (January 2013 to December 2017) were searched in PubMed, Cochrane Central Register of Controlled Trials, and EMBASE. Trial registration records identified in the publications and trial registries were classified into prospectively registered, retrospectively registered, or unregistered. Primary outcomes were identified and the direction of the results was judged as statistically significant (positive) or statistically nonsignificant (negative). We compared registered and published primary outcomes to assess outcome reporting bias and assessed whether discrepancies favored statistically significant outcomes. Frequency and strategies of spin in published reports with statistically nonsignificant results for primary outcomes were then identified. We also analyzed whether the trial registration status was associated with spin and quality of methodological factors.ResultsOf the 322 included RCTs, 41.9% (n = 135) were prospectively registered. Among 64 studies that were prospectively registered and specified primary outcomes, 25 trials had the discrepancies between the registered and published primary outcomes and 60% of them (15 trials) favored the statistically significant findings. Among 169 studies that specified primary outcomes, trial registration status was not associated with the direction of results, i.e., statistically significant or not. Spin was identified in 56.4% out of 78 studies with statistically nonsignificant primary outcomes and claiming efficacy with no consideration of statistically nonsignificant primary outcomes was the most common strategy for spin. Trial registration status was not statistically different between studies with and without spin.ConclusionWhile trial registration seemed to have improved over time, primary outcomes in registered records and publications were often inconsistent, tending to favor statistically significant findings and spin was common in studies with statistically nonsignificant primary outcomes. Journal editors and researchers in this field should be alerted to still prevalent reporting bias and spin.

show abstract

Registered trials report less beneficial treatment effects than unregistered ones: a meta-epidemiological study in orthodontics

Abstract: Signs of bias from lack of trial protocol registration were found with nonregistered trials reporting more beneficial intervention effects than registered ones. Caution is warranted by the interpretation of nonregistered randomized trials or systematic reviews thereof.

Cited by 29 publications

References 37 publications

Making ERP Research More Transparent: Guidelines for Preregistration

Making ERP Research More Transparent: Guidelines for Preregistration

Does Mindfulness Meditation Training Enhance Executive Control? A Systematic Review and Meta-Analysis of Randomized Controlled Trials in Adults

Trial registration as a safeguard against outcome reporting bias and spin? A case study of randomized controlled trials of acupuncture

Contact Info

Product

Resources

About