External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)

Abstract

BackgroundThousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews.MethodologyAMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently.ConclusionsThe sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearson's R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearson's R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were −0.19±1.38. This translates to a minimum detectable difference between reviews of 0.64 ‘AMSTAR points’. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews.

Reporting of Systematic Reviews and Meta-Analyses- A Bibliometric Analysis of Research Publications in Palliative Care Journals

“…The present study was not intended to perform a quality appraisal of the published systematic reviews, but future studies using AMSTAR-a quality measurement instrument 117 or its revised version-R-AMSTAR 118 could be performed to evaluate the systematic reviews in palliative care. These two tools have established reliability and validity 119,120 and previous such reports using AMSTAR tool were published in Indian medical journals. 121 More than one half of hospice and palliative care studies were reported in just 43 journals which were not specialist palliative and hospice care journals.…”

Appropriate Use Criteria for Somatostatin Receptor PET Imaging in Neuroendocrine Tumors

“…The quality (based on the risk of bias) of each study was categorized as "good," "fair," or "poor" using the U.S. Preventive Services Task Force criteria for randomized trials and cohort studies (33), Quality Assessment of Diagnostic Accuracy Studies-2 for diagnostic accuracy studies (34), and Assessment of Multiple Systematic Reviews for systematic reviews (35). The strength of overall evidence was graded as high, moderate, low, or very low using methods based on quality of evidence, consistency, directness, precision, and reporting bias.…”

Developing a spinal cord injury research strategy using a structured process of evidence review and stakeholder dialogue. Part I: rapid review of SCI prioritisation literature

“…Eligible quantitative SRs were critically appraised using the AMSTAR tool (http://amstar.ca/), an 11-item tool with well-established validity and reliability that is extensively used to evaluate quantitative SRs. [20][21][22] Critical appraisal of qualitative studies is an area of ongoing debate, and currently there is no consensus on an appropriate critical appraisal tool for qualitative research, and researchers are advised to choose a tool specific to this research paradigm. 23 Therefore, eligible qualitative SRs were critically appraised using five criteria designed to evaluate the rigor of qualitative reviews.…”