Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews

Saric, Frano; Barčot, Ognjen; Puljak, Livia

doi:10.1016/j.jclinepi.2019.04.007

Cited by 32 publications

(24 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To reduce bias, we made independent assessments by two authors for each analyzed domain and sub-domain. Additionally, we included in the analysis only the first four domains of the Cochrane RoB tool, because instructions from the Cochrane Handbook for these four domains are better characterized compared to the remaining three domains [ 11 – 13 ].…”

Section: Discussionmentioning

confidence: 99%

“…However, RoB assessments made by authors of published systematic reviews should not be taken at the face value, as we have shown in multiple studies that RoB assessments in many Cochrane reviews were inadequate and inconsistent [7][8][9][10][11][12][13][14]. Due to the specificities of surgical trials, we hypothesized that assessments of surgical trials may be more accurate and more consistent, compared to RCTs of nonsurgical interventions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Barčot

Borić

Došenović

et al. 2020

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Barčot

Borić

Došenović

et al. 2020

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, it is possible that the way the Cochrane authors judge RoB for sequence generation is highly variable. We have already proved this for RoB domains for allocation concealment [6], blinding of participants and personnel [7], incomplete outcome data [8], selective reporting [9], and other bias [10]. In that case, results presented in the study of Dechartres et al [5] or similar studies would not be based on consistent ratings of RoB in Cochrane reviews, and improvements shown for certain RoB domains could be misleading.…”

Section: Introductionmentioning

confidence: 99%

Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

Barčot

Borić

Peričić

et al. 2019

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

Background Assessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews. Cochrane systematic reviews appraise RoB of randomised controlled trials (RCTs) with the Cochrane RoB tool. Detailed instructions for using the Cochrane RoB tool are provided in the Cochrane Handbook for Systematic Reviews of Interventions (The Cochrane Handbook). The purpose of this study was to analyse whether Cochrane authors use adequate judgments about the RoB for random sequence generation of RCTs included in Cochrane reviews. Methods We extracted authors’ judgments (high, low or unclear RoB) and supports for judgments (comments accompanying judgments which explain the rationale for a judgment) for random sequence generation of included RCTs from RoB tables of Cochrane reviews using automated data scraping. We categorised all supporting comments, analysed the number and type of various supporting comments and assessed adequacy of RoB judgment for randomisation in line with recommendations from the Cochrane Handbook. Results We analysed 10,103 RCTs that were included in 704 Cochrane reviews. For 5,706 RCTs, randomisation was not described, but for the remaining RCTs, it was indicated that randomisation was performed using computer/software/internet ( N = 2,850), random number table ( N = 883), mechanical method ( N = 359) or it was incomplete/inappropriate ( N = 305). Overall, 1,220/10,103 trials (12%) did not have a RoB judgment in line with Cochrane Handbook guidance about randomisation. The highest proportion of misjudgements was found for trials with high RoB (28%), followed by those with low (20%) or unclear (3%). Therefore, one in eight judgments for the analysed domain in Cochrane reviews was not in line with Cochrane Handbook, and one in four if the judgment was "high risk". Conclusion Authors of Cochrane reviews often make judgments about the RoB related to random sequence generation that are not in line with instructions given in the Cochrane Handbook, which compromises the reliability of the systematic reviews. Our results can help authors of both Cochrane and non-Cochrane reviews which use Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials. Electronic supplementary material The online version of this article (10.1186/s12874-019-0804-y) contains supplementary material, which is available to authorized users.

show abstract

“…It has been shown previously that authors frequently make erroneous judgments (i.e., judgments that were not in line with the accompanying comment), and thus, not in line with recommendations available in the Cochrane Handbook. [25][26][27] Therefore, inadequate reporting of Cochrane risk of bias tool prevents readers to verify accuracy of authors' judgments.…”

Section: Discussionmentioning

confidence: 99%

Risk of Bias and Quality of Reporting in Colon and Rectal Cancer Systematic Reviews Cited by National Comprehensive Cancer Network Guidelines

et al. 2020

View full text Add to dashboard Cite

INTRODUCTION: Given the changing landscape of colorectal cancer, systematic reviews are likely to play a key role in advancing the understanding of prevention, diagnosis, and treatment. METHODS: We conducted a cross-sectional investigation of the risk of bias and reporting quality of systematic reviews referenced by colon and rectal cancer National Comprehensive Cancer Network (NCCN) guidelines. We used two widely accepted tools: Risk of Bias in Systematic reviews (ROBIS) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). RESULTS: Using ROBIS, only 3 (4.8%) systematic reviews were judged with low risk of bias, 35 (55.6%) systematic reviews were judged with unclear risk of bias, and 25 (39.7%) systematic reviews were judged with high risk of bias. Across all systematic reviews, the individual bias domains at the highest risk of bias were domains 1 (protocol and eligibility criteria) and 2 (methods to identify and select studies). Across all studies, the median adherence to PRISMA was 74.1% (IQR 69.2-80.0%), corresponding to approximately 20 of 27 items. CONCLUSIONS: Systematic reviews cited in NCCN guidelines for colon and rectal cancer are frequently at unclear or high risk of bias and do not report key systematic review items that are important for the critical appraisal of results.

show abstract

Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews

Cited by 32 publications

References 11 publications

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

Risk of Bias and Quality of Reporting in Colon and Rectal Cancer Systematic Reviews Cited by National Comprehensive Cancer Network Guidelines

Contact Info

Product

Resources

About