Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study

Abstract

BackgroundAssessing methodological quality of primary studies is an essential component of systematic reviews. Following a systematic review which used a domain based system [United States Preventative Services Task Force (USPSTF)] to assess methodological quality, a commonly used numerical rating scale (Downs and Black) was also used to evaluate the included studies and comparisons were made between quality ratings assigned using the two different methods. Both tools were used to assess the 20 randomized and quasi-randomized controlled trials examining an exercise intervention for chronic musculoskeletal pain which were included in the review. Inter-rater reliability and levels of agreement were determined using intraclass correlation coefficients (ICC). Influence of quality on pooled effect size was examined by calculating the between group standardized mean difference (SMD).ResultsInter-rater reliability indicated at least substantial levels of agreement for the USPSTF system (ICC 0.85; 95% CI 0.66, 0.94) and Downs and Black scale (ICC 0.94; 95% CI 0.84, 0.97). Overall level of agreement between tools (ICC 0.80; 95% CI 0.57, 0.92) was also good. However, the USPSTF system identified a number of studies (n = 3/20) as “poor” due to potential risks of bias. Analysis revealed substantially greater pooled effect sizes in these studies (SMD −2.51; 95% CI −4.21, −0.82) compared to those rated as “fair” (SMD −0.45; 95% CI −0.65, −0.25) or “good” (SMD −0.38; 95% CI −0.69, −0.08).ConclusionsIn this example, use of a numerical rating scale failed to identify studies at increased risk of bias, and could have potentially led to imprecise estimates of treatment effect. Although based on a small number of included studies within an existing systematic review, we found the domain based system provided a more structured framework by which qualitative decisions concerning overall quality could be made, and was useful for detecting potential sources of bias in the available evidence.Electronic supplementary materialThe online version of this article (doi:10.1186/s13104-015-1181-1) contains supplementary material, which is available to authorized users.

Reducing Clostridium difficile in the Inpatient Setting: A Systematic Review of the Adherence to and Effectiveness of C. difficile Prevention Bundles

“…23,24 This tool contains 27 dichotomous items regarding reporting, external validity, bias, confounding, and power. The maximum possible score of the modified instrument is 28, with the assessment of power modified from a 0–5 scale to 0–1.…”

Screening and evaluation tools of dysphagia in children with neuromuscular diseases: a systematic review

“…The same two investigators applied the quality index developed by Downs and Black for assessing methodological quality and bias . This tool comprises 27 questions relating to study description and external and internal validity, with a total maximum score of 28 . The tool was identified in a review by the Health Technology Group as one of the most appropriate for the evaluation of non‐randomized controlled trials in systematic reviews .…”

“…The tool was identified in a review by the Health Technology Group as one of the most appropriate for the evaluation of non‐randomized controlled trials in systematic reviews . As described by O'Connor et al, each study was assigned a grade of ‘excellent’ (24–28 points), ‘good’ (19–23 points), ‘fair’ (14–18 points), or ‘poor’ (<14 points) . Authors of included studies were contacted when necessary to obtain additional information.…”

Health effects of exposure to second- and third-hand marijuana smoke: a systematic review

“…A total score of 24-28 points = excellent, 19-23 points = good, 14-18 points = fair, less than 14 points = poor. 18 Research discomfort. 23 As a result, active smokers ceased smoking when they otherwise would have continued.…”