Guidance to best tools and practices for systematic reviews

Abstract

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

A timely review for systematic reviews

“…A timely manuscript in this month's issue of JBI Evidence Synthesis 1 offers an opportunity for reviewers and readers to take stock of and reflect on where the field of evidence synthesis, and organizational directives applicable to the field, including those from JBI, have arrived to date. Kolaski et al 1 highlight current deficiencies and confusion across terminology, methods of synthesis, and the application of the available methods by review authors-all of which rightfully cast doubt on the trustworthiness of many systematic reviews and question their authoritative claim to most appropriately guide decision-making. 1 Noteworthy among these issues, Kolaski et al 1 identify problems associated with the classification of primary study designs and the varying taxonomies and algorithms available to assist with classification.…”

“…Kolaski et al 1 highlight current deficiencies and confusion across terminology, methods of synthesis, and the application of the available methods by review authors-all of which rightfully cast doubt on the trustworthiness of many systematic reviews and question their authoritative claim to most appropriately guide decision-making. 1 Noteworthy among these issues, Kolaski et al 1 identify problems associated with the classification of primary study designs and the varying taxonomies and algorithms available to assist with classification. This issue is not exclusive to primary research; it is also apparent at the secondary research level, where the waters are further muddied by evolving methodologies of synthesis that continue to emerge.…”

“…Such confusion can cause errors in the conduct of these demanding and often complex research undertakings to the extent that many published reviews are flawed. 1 Considering this sobering realization, we encourage knowledge users to be vigilant when reading and interpreting the results of systematic reviews and to apply similar mechanisms of critique as they would to the results of any research. Despite all of this, when completed according to the best practices in the field, the systematic review can live up to its lofty expectation to inform the way forward to improved outcomes in health.…”

“…A timely manuscript in this month’s issue of JBI Evidence Synthesis 1 offers an opportunity for reviewers and readers to take stock of and reflect on where the field of evidence synthesis, and organizational directives applicable to the field, including those from JBI, have arrived to date. Kolaski et al 1 .…”

“…An additional concern the authors identify is that of redundant reviews, that is, reviews that overlap and may be deemed as wasteful and unnecessary 1 . This topic has been discussed repeatedly in the literature, most recently by Puljak and Lund 3 .…”