Assessing risk of bias judgments for blinding of outcome assessors in Cochrane reviews

Borić

Došenović

et al. 2020

BMC Med Res Methodol

Self Cite

Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

Section: Discussionmentioning

confidence: 73%

Section: Methodsmentioning

confidence: 99%

Section: Inclusion and Exclusion Criteriamentioning

confidence: 99%

See 1 more Smart Citation

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Borić

Došenović

et al. 2020

BMC Med Res Methodol

Self Cite

“…CDSR was searched for all reviews of randomized controlled trials (RCTs) of interventions (or both RCTs and non-randomized studies, but we analyzed RoB assessments only for RCTs) published from July 2015 to June 2016. This was a large, one-year convenient sample based on our previous studies [8,11,12], four years after the introduction of the 2011 RoB tool when it is expected from the review authors to have adopted the new methodology (tool). An advanced search option was used to limit results to content type and publication date.…”

Section: Inclusion and Exclusion Criteriamentioning

confidence: 99%

“…More specifically, in the performance bias domain, the overall proportion of RoB judgments following the recommendations from Cochrane Handbook (adequate judgments) was 73.6%, and the main error in reported RoB judgments was the presumption of healthcare providers being adequately blinded [11]. In the detection bias domain, the frequency of adequate judgments was 77.9%, and the main error was the improper categorization of outcomes (subjective vs. objective) [12]. Furthermore, we noticed that Cochrane authors still frequently use the joint domain for blinding of key individuals by making modifications to the 2011 Cochrane RoB tool, even though the tool contains two distinct blinding domains.…”

mentioning

confidence: 99%

Assessing the risk of performance and detection bias in Cochrane reviews as a joint domain is less accurate compared to two separate domains

Borić

Došenović

et al. 2021

BMC Med Res Methodol

Self Cite

Background Initially, the Cochrane risk of bias (RoB) tool had a domain for “blinding of participants, personnel and outcome assessors”. In the 2011 tool, the assessment of blinding was split into two domains: blinding of participants and personnel (performance bias) and blinding of outcome assessors (detection bias). The aims of this study were twofold; first, to analyze the frequency of usage of the joint blinding domain (a single domain for performance and detection bias), and second, to assess the proportion of adequate assessments made in the joint versus single RoB domains for blinding by comparing whether authors’ RoB judgments were supported by explanatory comments in line with the Cochrane Handbook recommendations. Methods We extracted information about the assessment of blinding from RoB tables (judgment, comment, and whether it was specified which outcome type; e.g., objective, subjective) of 729 Cochrane reviews published in 2015-2016. In the Cochrane RoB tool, judgment (low, unclear or high risk) needs to be accompanied by a transparent comment, in which authors provide a summary justifying RoB judgment, to ensure transparency in how these judgments were reached. We reassessed RoB based on the supporting comments reported in Cochrane RoB tables, in line with instructions from the Cochrane Handbook. Then, we compared our new assessments to judgments made by Cochrane authors. We compared the frequency of adequate judgments in reviews with two separate domains for blinding versus those with a joint domain for blinding. Results The total number of assessments for performance bias was 6918, with 8656 for detection bias and 3169 for the joint domain. The frequency of adequate assessments was 74% for performance bias, 78% for detection bias, and 59% for the joint domain. The lowest frequency of adequate assessments was found when Cochrane authors judged low risk – 47% in performance bias, 62% in detection bias, and 31% in the joint domain. The joint domain and detection bias domain had a similar proportion of specified outcome types (17% and 18%, respectively). Conclusions Splitting joint RoB assessment about blinding into two domains was justified because the frequency of adequate judgments was higher in separate domains. Specification of outcome types in RoB domains should be further scrutinized.

Enhanced access to recommendations from the Cochrane Handbook for improving authors' judgments about risk of bias: A randomized controlled trial

Research Synthesis Methods

Ivanda

Buljan

et al. 2021

Self Cite

This randomized controlled trial (RCT) aimed to test the efficacy of enhanced access to Cochrane Handbook (Handbook) recommendations for judging the 2011 Cochrane risk of bias (RoB) domains for improving the adequacy of RoB judgments. Parallel‐group RCT with a 1:1 allocation ratio (N = 2271 per group) was conducted. Eligible participants were corresponding authors of all published Cochrane reviews and protocols. After allocation by a random number generator, participants received 20 scenarios for assessing RoB. The intervention group was shown tables from the Handbook with instructions for assessing 2011 RoB tool together with scenarios they were supposed to assess—enhanced access to the Handbook. The control group was shown only a general link to the Handbook. The primary outcome was the proportion of participants that made an adequate judgment of RoB scenarios for analyzed domains. There were 240 responses out of 2020 delivered e‐mail invitations in the intervention and 197/2254 in the control group. Only five participants from the intervention group judged RoB adequately in all the 20 scenarios and no one in the control group. The proportion of participants who adequately assessed all the scenarios within a domain was significantly higher in the intervention than in the control group. The frequency of adequate RoB judgments was 7.1% (95% CI: 5.0–9.3%, p < 0.001) higher in the intervention group (76.2%) than in the control group (69.0%). The enhanced access yields more adequate RoB assessments and could be incorporated in software supporting the RoB tool.