Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study

Abstract

Objective To simulate possible changes in systematic review results if rapid review methods were used. Study Design and Setting We recalculated meta-analyses for binary primary outcomes in Cochrane systematic reviews, simulating rapid review methods. We simulated searching only PubMed, excluding older articles (5, 7, 10, 15, and 20 years before the search date), excluding smaller trials (<50, <100, and <200 participants), and using the largest trial only. We examined percentage changes in pooled odds ratios (ORs) (classed as no important change [<5%], small [<20%], moderate [<30%], or large [≥30%]), statistical significance, and biases observed using rapid methods. Results Two thousand five hundred and twelve systematic reviews (16,088 studies) were included. Rapid methods resulted in the loss of all data in 3.7–44.7% of meta-analyses. Searching only PubMed had the smallest risk of changed ORs (19% [477/2,512] were small changes or greater; 10% [260/2,512] were moderate or greater). Changes in ORs varied substantially with each rapid review method; 8.4–21.3% were small, 1.9–8.8% were moderate, and 4.7–34.1% were large. Changes in statistical significance occurred in 6.5–38.6% of meta-analyses. Changes from significant to nonsignificant were most common (2.1–13.7% meta-analyses). We found no evidence of bias with any rapid review method. Conclusion Searching PubMed only might be considered where a ∼10% risk of the primary outcome OR changing by >20% could be tolerated. This could be the case in scoping reviews, resource limitation, or where syntheses are needed urgently. Other situations, such as clinical guidelines and regulatory decisions, favor more comprehensive systematic review methods.

You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

“…For the stopping prediction task, CLF could significantly reduce the cost of screening with tolerable losses in recall. The results of the stopping prediction task demonstrate the applicability of CLF to specific systematic reviews where total recall is not essential, such as in rapid reviews [34].…”

Assessing the risk of bias in choice of search sources for environmental meta‐analyses

“…Records' titles and/or Digital Object Identifiers (DOIs) were used to check the index status of each study in the platforms. Where studies were indexed in a platforms, we treated them as retrieved by searches of that platform because our interest was not in the performance of search strategies actually used, rather the risk of sole reliance on studies that were indexed in the platforms (hence findable) 19 . This approach was chosen because applying the original search strings might miss studies indexed in the platforms 19 .…”

“…Where studies were indexed in a platforms, we treated them as retrieved by searches of that platform because our interest was not in the performance of search strategies actually used, rather the risk of sole reliance on studies that were indexed in the platforms (hence findable) 19 . This approach was chosen because applying the original search strings might miss studies indexed in the platforms 19 . To enable subgroup analyses, we also checked study sources and classified those into: (a) commercially published article; and (b) grey literature and unpublished data by applying the Luxembourg definition of grey literature: “manifold document types produced on all levels of government, academics, business, and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by libraries and institutional repositories, but not controlled by commercial publishers” 25 .…”

“…Note bias and random error cannot often be distinguished, thus we assess the risk of availability bias rather than testing presence or absence of availability bias (see Reference 18 for the concept of “risk of bias”). To date, no studies assessed the impacts of search sources on environmental meta‐analytical estimates 19 . Since the number of environmental meta‐analyses are increasing rapidly, 7,20 and are increasingly used to inform policy and practice, the assessment of risk of bias in search sources is a critical element of review conduct.…”

Impact of racism and discrimination on the physical and mental health outcomes among Aboriginal and Torres Strait Islander peoples living in Australia: a protocol for a scoping review

“…The search will be conducted in English and will include studies published between the 01 st of January of 2000 and June 2020. The rationale behind the search from year 2000 to 2020 is ndings from a meta-analysis study which showed that searches including articles from the last 20 years did not result in the loss of any primary study [35].…”