Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study

Abstract

BackgroundSystematic reviews offer the most reliable and valid support for health policy decision-making, patient information, and guideline development. However, they are labor intensive and frequently take longer than 1 year to complete. Consequently, they often do not meet the needs of those who need to make decisions quickly. Rapid reviews have therefore become a pragmatic alternative to systematic reviews. They are knowledge syntheses that abbreviate certain methodological aspects of systematic reviews to produce information more quickly. Methodological shortcuts often take place in literature identification. A potential drawback is less reliable results. To date, the impact of abbreviated searches on estimates of treatment effects and subsequent conclusions has not been analyzed systematically across multiple bodies of evidence. We aim to answer the research question: Do bodies of evidence that are based on abbreviated literature searches lead to different conclusions about benefits and harms of interventions compared with bodies of evidence that are based on comprehensive, systematic literature searches?MethodsWe will use a non-inferiority and meta-epidemiologic design. The primary outcome is the proportion of discordant conclusions based on different search approaches. Drawing of a pool of Cochrane reports published between 2012 and 2016, we will randomly select 60 reports. Eligible reports are those that present a summary-of-findings table, draw a clear conclusion, present data for meta-analyses, and document the search strategy clearly. We will conduct several abbreviated searches to detect whether included studies in these Cochrane reviews could be detected. If searches could not detect all studies, we will revise the original summary-of-findings table and ask review authors whether the missed evidence would change conclusions of their report. We will determine the proportion of discordant conclusions for each abbreviated search approach. We will consider an abbreviated search as non-inferior if the lower limit of the 95% confidence interval of the proportion of discordant conclusions is below the non-inferiority margin, which is determined based on results of a survey for clinical and public health scenarios.DiscussionThis will be the first study to assess whether the reduced sensitivity of abbreviated searches has an impact on conclusions across multiple bodies of evidence, not only on effect estimates.

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

“…Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ].…”

Combining abbreviated literature searches with single-reviewer screening: three case studies of rapid reviews

“…The primary outcome of this study was the proportion of concordant conclusions between the rapid reviews and the Cochrane reviews. To address this outcome, we applied a method by Nussbaumer-Streit et al [23]. We considered studies as identified if at least one publication of a relevant study had been included.…”

Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey

“…The impact of studies that were missed on effect estimates of meta-analyses, however, was small [ 24 ]. An ongoing methods project uses a non-inferiority and meta-epidemiological approach to assess whether the reduced sensitivity of abbreviated searches has an impact on conclusions of Cochrane reviews, not only on effect estimates [ 25 ].…”