Resuming the discussion of AMSTAR: What can (should) be made better?

Abstract

BackgroundEvidence syntheses, and in particular systematic reviews (SRs), have become one of the cornerstones of evidence-based health care. The Assessment of Multiple Systematic Reviews (AMSTAR) tool has become the most widely used tool for investigating the methodological quality of SRs and is currently undergoing revision. The objective of this paper is to present insights, challenges and potential solutions from the point of view of a group of assessors, while referring to earlier methodological discussions and debates with respect to AMSTAR.DiscussionOne major drawback of AMSTAR is that it relies heavily on reporting quality rather than on methodological quality. This can be found in several items. Furthermore, it should be acknowledged that there are now new methods and procedures that did not exist when AMSTAR was developed. For example, the note to item 1 should now refer to the International Prospective Register of Ongoing Systematic Reviews (PROSPERO). Furthermore, item 3 should consider the definition of hand-searching, as the process of reviewing conference proceedings using the search function (e.g. in Microsoft Word or in a PDF file) does not meet the definition set out by the Cochrane Collaboration. Moreover, methods for assessing the quality of the body of evidence have evolved since AMSTAR was developed and should be incorporated into a revised AMSTAR tool.SummaryPotential solutions are presented for each AMSTAR item with the aim of allowing a more thorough assessment of SRs. As the AMSTAR tool is currently undergoing further development, our paper hopes to add to preceding discussions and papers regarding this tool and stimulate further discussion.

Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions

“…Despite the uncertainty regarding use of summary scores, there is a precedent for calculating and reporting overall AMSTAR assessments both in empirical studies assessing measurement properties of AMSTAR [20, 21, 29–32, 54–56] and in overviews of healthcare interventions [16, 18, 35, 36, 57–71], and incorporating overall quality of SRs into results of overviews has not been found to alter overview results [72]. Other potential limitations of AMSTAR may include difficulty meaningfully differentiating between several of the response options (“no”, “not applicable”, and “can’t answer”) and difficulty answering multi-part questions when only some criteria are met [44, 45]. There are also no questions in the AMSTAR tool examining whether appropriate methods were used in SRs to assess the quality of a body of evidence or to conduct subgroup and/or sensitivity analyses [44, 45], and in the context of overviews AMSTAR cannot capture potentially important differences in comprehensiveness and recency of searches across SRs.…”

“…Other potential limitations of AMSTAR may include difficulty meaningfully differentiating between several of the response options (“no”, “not applicable”, and “can’t answer”) and difficulty answering multi-part questions when only some criteria are met [44, 45]. There are also no questions in the AMSTAR tool examining whether appropriate methods were used in SRs to assess the quality of a body of evidence or to conduct subgroup and/or sensitivity analyses [44, 45], and in the context of overviews AMSTAR cannot capture potentially important differences in comprehensiveness and recency of searches across SRs. As previously mentioned, the AMSTAR tool may also assess aspects related to quality of reporting as opposed to methodological quality [44–46].…”

Efficacy of coloured overlays and lenses for treating reading difficulty: an overview of systematic reviews

“…During this process, a new checklist, AMSTAR 2, was published. Since the latter was developed to overcome limitations of the original version, the new checklist was applied instead. Note that the PROSPERO protocol for this review indicates that AMSTAR would be used and the register includes additional information stating that the AMSTAR 2 was used instead.…”

Simulation Modelling in Healthcare: An Umbrella Review of Systematic Literature Reviews

“…The AMSTAR tool consists of 11 key questions that have adequate face and content validity to measure quality of systematic reviews effectively [6]. However no instrument currently exists to assess the quality of methodology reviews.…”

“…We selected the AMSTAR (a measurement tool used to assess systematic reviews) checklist from [5] which is widely recognised as a way of evaluating reviews [6]. The AMSTAR tool consists of 11 key questions that have adequate face and content validity to measure quality of systematic reviews effectively [6].…”