Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability

Abstract

Background An important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews are considered golden standard regarding systematic review methodology, but Cochrane’s instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors’ assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR). Methods We analyzed Cochrane reviews published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many Cochrane reviews had different judgments for the same supporting explanations. Results In the main analysis we included 10,292 judgments and supporting explanations for attrition bias from 729 Cochrane reviews. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of Cochrane review authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same Cochrane review. Conclusion We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane’s risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare. Electronic supplementary material The online version of this article (10.1186/s12874-019-0717-9) contains supplementary material, which is available to authorized users.

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

“…To reduce bias, we made independent assessments by two authors for each analyzed domain and sub-domain. Additionally, we included in the analysis only the first four domains of the Cochrane RoB tool, because instructions from the Cochrane Handbook for these four domains are better characterized compared to the remaining three domains [ 11 – 13 ].…”

“…However, RoB assessments made by authors of published systematic reviews should not be taken at the face value, as we have shown in multiple studies that RoB assessments in many Cochrane reviews were inadequate and inconsistent [7][8][9][10][11][12][13][14]. Due to the specificities of surgical trials, we hypothesized that assessments of surgical trials may be more accurate and more consistent, compared to RCTs of nonsurgical interventions.…”

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

“…A limitation of our study is that we considered only dichotomous outcome data; methods for handling missing continuous data are different and our findings might not be generalisable to systematic reviews of continuous outcomes 2930. Our sample consisted of systematic reviews that were published in 2012 and these might not reflect more current reviews; however, recent surveys have found that the reporting, handling, and assessment of risk of bias in relation to missing data has not improved over the date of our search 7313233. For further confirmation that current practice is unlikely to have changed, we assessed the reporting and handling of missing data in a sample of recently published systematic reviews.…”

“… 29 30 Our sample consisted of systematic reviews that were published in 2012 and these might not reflect more current reviews; however, recent surveys have found that the reporting, handling, and assessment of risk of bias in relation to missing data has not improved over the date of our search. 7 31 32 33 For further confirmation that current practice is unlikely to have changed, we assessed the reporting and handling of missing data in a sample of recently published systematic reviews. To make our selection of a sample of systematic reviews reproducible, we ordered systematic reviews published from January 2020 in the chronological order of their publication and selected the first 15 that met our eligibility criteria.…”

Efficacy and safety of modified-release paracetamol for acute and chronic pain: a systematic review protocol

“…Cochrane risk of bias (RoB) tool, with seven dimensions, including sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting and other potential threats to study validity will be used. We will assess RoB in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions22 to avoid mistakes review authors frequently make when conducting RoB assessments with this tool 23–27. For each study, we will create a RoB domain, which will have a RoB judgement for each domain (high, low or unclear risk) and an accompanying supporting comment to explain the RoB judgement.…”