Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study

Abstract

BackgroundSelective reporting is included as a core domain of Cochrane’s tool for assessing risk of bias in randomised trials. There has been no evaluation of review authors’ use of this domain. We aimed to evaluate assessments of selective reporting in a cross-section of Cochrane reviews and to outline areas for improvement.MethodsWe obtained data on selective reporting judgements for 8434 studies included in 586 Cochrane reviews published from issue 1–8, 2015. One author classified the reasons for judgements of high risk of selective reporting bias. We randomly selected 100 reviews with at least one trial rated at high risk of outcome non-reporting bias (non-/partial reporting of an outcome on the basis of its results). One author recorded whether the authors of these reviews incorporated the selective reporting assessment when interpreting results.ResultsOf the 8434 studies, 1055 (13 %) were rated at high risk of bias on the selective reporting domain. The most common reason was concern about outcome non-reporting bias. Few studies were rated at high risk because of concerns about bias in selection of the reported result (e.g. reporting of only a subset of measurements, analysis methods or subsets of the data that were pre-specified). Review authors often specified in the risk of bias tables the study outcomes that were not reported (84 % of studies) but less frequently specified the outcomes that were partially reported (61 % of studies). At least one study was rated at high risk of outcome non-reporting bias in 31 % of reviews. In the random sample of these reviews, only 30 % incorporated this information when interpreting results, by acknowledging that the synthesis of an outcome was missing data that were not/partially reported.ConclusionsOur audit of user practice in Cochrane reviews suggests that the assessment of selective reporting in the current risk of bias tool does not work well. It is not always clear which outcomes were selectively reported or what the corresponding risk of bias is in the synthesis with missing outcome data. New tools that will make it easier for reviewers to convey this information are being developed.Electronic supplementary materialThe online version of this article (doi:10.1186/s13643-016-0289-2) contains supplementary material, which is available to authorized users.

<p>Depression And Anxiety In Patients With Juvenile Idiopathic Arthritis: Current Insights And Impact On Quality Of Life, A Systematic Review</p>

“…The remaining 799 articles were organized in Rayyan Qatar Computing Research Institute (QCRI), a web-based app for systematic reviews. 22 All decisions for study inclusions and exclusions were taken by consensus based on how well the study fit the inclusion criteria. Each abstract was screened by two of three authors (D.F., M.R., and T.R.)…”

Component studies of psychological treatments of adult depression: A systematic review and meta-analysis

“…This tool assesses possible sources of bias in randomized trials, including the adequate generation of allocation sequence; the concealment of allocation to conditions; the prevention of knowledge of the allocated intervention (masking of assessors; when only self-report measures were used for the assessment of depression we rated this as positive as well); and dealing with incomplete outcome data (this was assessed as positive when intention-to-treat analyses were conducted, meaning that all randomized patients were included in the analyses). We did not rate selective outcome reporting, because there are important flaws in its assessment (Page & Higgins, 2016), such as the conflation of nonreporting with selective reporting of outcomes, and because most psychotherapy trials are still not prospectively registered (Bradley, Rucklidge, & Mulder, 2017).…”

Assessing risk of bias in studies that evaluate health care interventions: recommendations in the misinformation age

“…results that are unfavourable to the experimental intervention are not reported) (18). This tool will guide users to consider risk of bias in a synthesis due to both selective publication of whole studies and selective non-reporting of outcomes within study reports, given that both practices lead to the same consequence: evidence missing from the synthesis (19). The tool will complement the ROBINS-I (12) M A N U S C R I P T A C C E P T E D ACCEPTED MANUSCRIPT 7 and RoB 2.0 (13) tools, which include a domain for assessing the risk of bias in selection of a fully reported result, but no mechanism to assess risk of bias due to selective non-reporting.…”