The judgement of biases included in the category “other bias” in Cochrane systematic reviews of interventions: a systematic survey

Abstract

Background Clinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the ‘other bias’ domain of Cochrane RoB tool. Methods We analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding ‘other bias’ from the RoB table and accompanying support for the judgment. We categorized different types of other bias. Results We analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had ‘other bias’ domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. Conclusion Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews. Electronic supplementary material The online version of this article (10.1186/s12874-019-0718-8) contains supplementary material, which is available to authorized users.

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

“…To reduce bias, we made independent assessments by two authors for each analyzed domain and sub-domain. Additionally, we included in the analysis only the first four domains of the Cochrane RoB tool, because instructions from the Cochrane Handbook for these four domains are better characterized compared to the remaining three domains [11][12][13].…”

“…However, RoB assessments made by authors of published systematic reviews should not be taken at the face value, as we have shown in multiple studies that RoB assessments in many Cochrane reviews were inadequate and inconsistent [7][8][9][10][11][12][13][14]. Due to the specificities of surgical trials, we hypothesized that assessments of surgical trials may be more accurate and more consistent, compared to RCTs of nonsurgical interventions.…”

Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

“…However, it is possible that the way the Cochrane authors judge RoB for sequence generation is highly variable. We have already proved this for RoB domains for allocation concealment [6], blinding of participants and personnel [7], incomplete outcome data [8], selective reporting [9], and other bias [10]. In that case, results presented in the study of Dechartres et al [5] or similar studies would not be based on consistent ratings of RoB in Cochrane reviews, and improvements shown for certain RoB domains could be misleading.…”

Efficacy and safety of modified-release paracetamol for acute and chronic pain: a systematic review protocol

“…Cochrane risk of bias (RoB) tool, with seven dimensions, including sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting and other potential threats to study validity will be used. We will assess RoB in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions22 to avoid mistakes review authors frequently make when conducting RoB assessments with this tool 23–27. For each study, we will create a RoB domain, which will have a RoB judgement for each domain (high, low or unclear risk) and an accompanying supporting comment to explain the RoB judgement.…”