Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data

Lunny, Carole; Brennan, Sue; Reid, Jane; McDonald, Steve; McKenzie, Joanne E.

doi:10.1016/j.jclinepi.2019.09.025

“…Overlap in meta‐reviews is a methodological issue that, if ignored or improperly handled, can lead to false assumptions about the evidence presented. As a result, some meta‐review methodologists have examined whether and how author teams have addressed this issue with fairly disappointing results, indicating authors are not taking this issue seriously enough . Thus, authors have proposed to develop a citation matrix and calculate the CCA.…”

Section: Discussionmentioning

confidence: 99%

“…Meta‐reviews can serve several purposes, including to compare and contrast the findings from systematic reviews and to provide a summary of the evidence and the quality of the evidence across these reviews . Along with the rapid growth of meta‐reviews, there are numerous methodological tools and considerations, many of which are still in the early phases of development or for which there are no established guidelines . Overlap of the primary‐level studies in meta‐reviews is one such area for consideration and an important one for evidence synthesis methodologists because it concerns the potential for non‐independence in the primary studies across similar reviews.…”

Section: Introductionmentioning

confidence: 99%

Examining overlap of included studies in meta‐reviews: Guidance for using the corrected covered area index

Hennessy

¹

,

Johnson

²

2019

Research Synthesis Methods

View full text Add to dashboard Cite

Overlap in meta-reviews results from the use of multiple identical primary studies in similar reviews. It is an important area for research synthesists because overlap indicates the degree to which reviews address the same or different literatures of primary research. Current guidelines to address overlap suggest that assessing and documenting the degree of overlap in primary studies, calculated via the corrected covered area (CCA) is a promising method. Yet, the CCA is a simple percentage of overlap and current guidelines do not detail ways that reviewers can use the CCA as a diagnostic tool while also comprehensively incorporating these findings into their conclusions. Furthermore, we maintain that meta-review teams must address non-independence via overlap more thoroughly than by simply estimating and reporting the CCA. Instead, we recommend and elaborate five steps to take when examining overlap, illustrating these steps through the use of an empirical example of primary study overlap in a recently conducted meta-review. This work helps to show that overlap of primary studies included in a meta-review is not necessarily a bias but often can be a benefit. We also highlight further areas of caution in this task and potential for the development of new tools to address nonindependence issues. K E Y W O R D Scitation matrix, corrected covered area, meta-review, overlap, overview

show abstract

“…There is insu cient empirical evidence, however, to fully understand how many of these methodological decisions may impact reviewer workload, the validity of results and conclusions of overviews of reviews, and their relevance for healthcare decision-makers [10]. Since there does not yet exist a minimum standard of conduct and reporting, published overviews of reviews use highly heterogeneous methodologies [9,10,13,38] and are often poorly and inconsistently reported [4,5,12]. The propagation of substandard overviews of reviews has the potential to undermine their legitimacy as an important tool for healthcare decision-making, and substantiates the urgent need to develop evidence-based conduct and reporting standards akin to what exists for systematic reviews [39,40].…”

Section: Discussionmentioning

confidence: 99%

“…Evaluation and recommendations for applying suggested methods is sparse [9,10], and evidence-based guidance for the complete and transparent reporting of overviews of reviews is lacking [11]. As a result, the conduct and reporting of overviews of reviews is often inadequate and inconsistent [4,5,12]. As the science of overviews of reviews continues to develop, authors will need to keep up-to-date with the latest methods research and reporting guidelines [13].…”

Section: Introductionmentioning

confidence: 99%

Guidance for Overviews of Reviews Continues to Accumulate, But Important Challenges Remain: A Scoping Review

Gates

¹

,

Gates

²

,

Guitard

³

et al. 2020

Preprint

0

View full text Add to dashboard Cite

Background. Overviews of reviews (overviews) provide an invaluable resource for healthcare decision-making by combining large volumes of systematic review (SR) data into a single synthesis. The production of high-quality overviews hinges on the availability of practical evidence-based guidance for conduct and reporting.Objectives. Within the broad purpose of informing the development of a reporting guideline for overviews, we aimed to provide an up-to-date map of existing guidance related to the conduct of overviews, and to identify common challenges that authors face when undertaking overviews.Methods. We updated a scoping review published in 2016 using the search methods that had produced the highest yield: ongoing reference tracking (2014 to March 2020 in Pubmed, Scopus, and Google Scholar), hand-searching conference proceedings and websites, and contacting authors of published overviews. Using a qualitative meta-summary approach, one reviewer extracted, organized, and summarized the guidance and challenges presented within the included documents. A second reviewer verified the data and synthesis.Results. We located 28 new guidance documents, for a total of 77 documents produced by 34 research groups. The new guidance helps to resolve some earlier identified challenges in the production of overviews. Important developments include strengthened guidance on handling primary study overlap at the study selection and analysis stages. Despite marked progress, several areas continue to be hampered by inconsistent or lacking guidance. There is ongoing debate about whether, when, and how supplemental primary studies should be included in overviews. Guidance remains scant on how to extract and use appraisals of quality of the primary studies within the included SRs and how to adapt GRADE methodology to overviews. The challenges that overview authors face are often related to the above-described steps in the process where evidence-based guidance is lacking or conflicting.Conclusion. The rising popularity of overviews has been accompanied by a steady accumulation of new, and sometimes conflicting, guidance. While recent guidance has helped to address some of the challenges that overview authors face, areas of uncertainty remain. Practical tools supported by empirical evidence are needed to assist authors with the many methodological decision points that are encountered in the production of overviews.

show abstract

“…Many methods to conduct the various stages of overviews of reviews have been suggested; however, much of the guidance is inconsistent, and evidence-based reporting guidance is lacking [9]. The relative lack of evidence and consistency in recommendations may underpin the inadequate and inconsistent conduct and reporting of overviews to date [4,5,10]. As the science of overviews of reviews continues to develop, authors will need to keep up-todate with the latest methods research and reporting guidelines [11].…”

Section: Introductionmentioning

confidence: 99%

Guidance for overviews of reviews continues to accumulate, but important challenges remain: A scoping review

Gates

¹

,

Gates

²

,

Guitard

³

et al. 2020

Preprint

View full text Add to dashboard Cite

Background. Overviews of reviews (overviews) provide an invaluable resource for healthcare decision-making by combining large volumes of systematic review (SR) data into a single synthesis. The production of high-quality overviews hinges on the availability of practical evidence-based guidance for conduct and reporting.Objectives. Within the broad purpose of informing the development of a reporting guideline for overviews, we aimed to provide an up-to-date map of existing guidance related to the conduct of overviews, and to identify common challenges that authors face when undertaking overviews.Methods. We updated a scoping review published in 2016 using the search methods that had produced the highest yield: ongoing reference tracking (2014 to March 2020 in Pubmed, Scopus, and Google Scholar), hand-searching conference proceedings and websites, and contacting authors of published overviews. Using a qualitative meta-summary approach, one reviewer extracted, organized, and summarized the guidance and challenges presented within the included documents. A second reviewer verified the data and synthesis.Results. We located 28 new guidance documents, for a total of 77 documents produced by 34 research groups. The new guidance helps to resolve some earlier identified challenges in the production of overviews. Important developments include strengthened guidance on handling primary study overlap at the study selection and analysis stages. Despite marked progress, several areas continue to be hampered by inconsistent or lacking guidance. There is ongoing debate about whether, when, and how supplemental primary studies should be included in overviews. Guidance remains scant on how to extract and use appraisals of quality of the primary studies within the included SRs and how to adapt GRADE methodology to overviews. The challenges that overview authors face are often related to the above-described steps in the process where evidence-based guidance is lacking or conflicting.Conclusion. The rising popularity of overviews has been accompanied by a steady accumulation of new, and sometimes conflicting, guidance. While recent guidance has helped to address some of the challenges that overview authors face, areas of uncertainty remain. Practical tools supported by empirical evidence are needed to assist authors with the many methodological decision points that are encountered in the production of overviews.

show abstract

Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data

Cited by 57 publications

References 25 publications

Examining overlap of included studies in meta‐reviews: Guidance for using the corrected covered area index

Examining overlap of included studies in meta‐reviews: Guidance for using the corrected covered area index

Guidance for Overviews of Reviews Continues to Accumulate, But Important Challenges Remain: A Scoping Review

Guidance for overviews of reviews continues to accumulate, but important challenges remain: A scoping review

Contact Info

Product

Resources

About