Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

Abstract

BackgroundThe increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.Methodology/Principal FindingsWe review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.ConclusionsRecent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

A global comprehensive review of economic interventions to prevent intimate partner violence and HIV risk behaviours

“…There is a widely known publication bias whereby positive results are much more likely to be published [90], which while somewhat mitigated by the inclusion of grey material, is unlikely to have been totally overcome. Additionally, as any quantitative evaluation was included, irrespective of the quality of the study design, comparing different studies to each other may misrepresent interventions’ true effectiveness.…”

Selective outcome reporting: telling and detecting true lies. The state of the science

“…The addition and removal of outcomes together with the sample size recalculation carry a high risk of bias. In fact, the prevalence of favourable results from the cohort of trials examined in the reported surveys is very high, and may reflect no true differences [18].…”

“…The availability of trial protocols made it evident that often the published reports do not correspond to the registered study protocol. The findings of the principal studies that compared outcomes between protocols and publications are summarised in Table 1 (freely adapted from Table 5 of Dwan et al [18]). In comparing trial publications to protocols, Dwan's systematic review finds that 40-62% of studies have at least one primary outcome that was changed, introduced, or omitted, and that outcomes that are statistically significant have higher odds of being fully reported (range of odds ratios: 2.2-4.7).…”

In Pursuit of Greater Reproducibility and Credibility of Early Clinical Biomarker Research

“…A related phenomenon is selective reporting of results within a study (e.g., only for certain outcome measures or subgroups among many examined), where usually those reported are statistically significant, especially in a desired or expected direction. Evidence for publication bias and selective reporting in clinical trials has been firmly established 9. For early clinical biomarker research, the potential for biases is greater due to lack of an organized system for study registration (analogous to ClinicalTrials.gov for clinical trials) and typical absence of comprehensive study protocols with prespecified statistical analysis plans.…”