Decision makers find it difficult to compare and select similar systematic reviews based on quality, methods and results: a cross-sectional survey

Abstract

Background Systematic reviews (SRs) are being published at an accelerated rate. Decision makers are often faced with the challenge of comparing and choosing between multiple SRs on the same topic. We surveyed individuals in the healthcare field to understand what criteria they use to compare and select one or more SRs from multiple on the same topic. Methods We developed a survey with 21 open and closed questions. We disseminated it through social media and professional networks. Results Of the 684 respondents, 25% were health practitioners, 9% policymakers, 39% researchers, and 13% were students. Policymakers, practitioners and researchers frequently sought out SRs (98.1%) as a source of evidence to inform decision making. They frequently (97.7%) found more than one SR on a given topic of interest to them. Half (50%) struggled to choose the most valid and trustworthy SR amongst multiple. These difficulties related to lack of time (55.2%), insufficient skills/ experience in quality appraisal (27.7%), and difficulty comparing different SRs (54.3%). Respondents compared SRs based on: relevance to their question of interest; its methodological quality, and recency of SR search date. Conclusions The exponential growth in the number of SRs leads to duplication of research on similar questions and compounds the problem of identifying which evidence is of the highest quality for decision makers. Failure to critically appraise and choose the highest quality SRs means that practice and policy decisions may not reflect the best evidence, the implementation of better intervention strategies is delayed, and patients may unduly suffer.

User experience of applying AMSTAR 2 to appraise systematic reviews of healthcare interventions: a commentary

“…AMSTAR 2 could be used to detect SRs with the lowest confidence ratings and exclude them from the pool of relevant SRs for policy development or health decision-making [39]. Such classification of SRs could be helpful for decision makers, who find it difficult to select appropriate SRs for their work [40]. Our recommendations could be considered to improve the usability of AMSTAR 2 for health decision-making.…”

Different Approaches to Appraising Systematic Reviews of Digital Interventions for Physical Activity Promotion Using AMSTAR 2 Tool: Cross-Sectional Study

“…The appraisals with these two items could be performed by less experienced users of AMSTAR 2 because presence of a review protocol and a list of excluded studies can be identified relatively fast and does not require as much methodological expertise as some other items on AMSTAR 2 (e.g., item 11 that requires a judgement of methods used in a meta-analysis). Since decision makers find it difficult to select appropriate SRs for their work [ 14 ], the appraisal approach with two critical items could assist with SR classification and selection for further work. For example, such an approach can be used when large numbers of SRs on a similar topic are available for their potential application in health decision-making.…”

Quality of systematic reviews on physiotherapy interventions for musculoskeletal disorders is critically low: a meta-epidemiological study

“…9–12 We have recently conducted a survey confirming that more than 50% of decision-makers have difficulty choosing the best evidence on a given topic: the methodological quality, the reputation of the authors and the journal, and the type of primary studies included were among the features thought to be important by respondents. 13…”