How to decide whether a systematic review is stable and not in need of updating: Analysis of Cochrane reviews

Abstract

It is challenging to keep systematic reviews (SR) current and updated. Cochrane designated some of its SRs as "stable," that is, not in need of updating. The issue of stabilizing an SR is an important in research synthesis, because it could help reduce research waste. The aim of this study was to analyze publicly available justifications for stabilizing a Cochrane review, with the ultimate goal of helping to make decisions about whether the update of any SR is warranted. Methods: We analyzed Cochrane reviews labeled as stable in Archie, Cochrane's system for managing the editorial/publishing process. From the "What's new" section of the reviews in the Cochrane Library, we extracted justification for stabilization. Results: We included 545 Cochrane reviews labeled in Archie as stable on October 28, 2019. The most common of the five reasons for stabilization was that "last search did not identify any potentially relevant studies likely to change conclusions" (N = 99; 18%), followed by "research area no longer active" (N = 86; 16%), "review is or will be superseded" (N = 41; 7.5%), "evidence is conclusive" (N=35; 6.4%), and "intervention no longer in general use" (N = 34; 6.2%). For the 269 (49%) Cochrane reviews, we considered that the justification for stabilization was not clearly described, that is, sufficiently informative. Conclusions: Cochrane reviews would benefit from more transparency and consistency in publicly available justifications for stabilizing reviews. Further work in this field will help make decisions about the futility of further research and deciding on enough evidence in the field of research synthesis.

When is the evidence conclusive? Analysis of systematic reviews for which Cochrane declared that conclusions will not change with further studies

“…Moreover, a lack of transparency in labeling the Cochrane reviews as stable will likely lead to increased research waste. 4 4.2 | Unclear criteria for deciding that there is enough evidence It has been reported before that few Cochrane protocols, less than 5%, specified analytic methods that would determine that there is enough evidence for stopping updating a review. Both, our earlier study and another report found that the majority of Cochrane reviews were stabilized for reasons other than not having enough evidence.…”

“…Both, our earlier study and another report found that the majority of Cochrane reviews were stabilized for reasons other than not having enough evidence. 4,11 However, the research community would benefit from clear criteria (i.e., decision-making algorithms) for conclusiveness of evidence.…”

“…If the authors know that there is enough evidence on the specific topic, based on a highquality SR and that SR results will not change the conclusions with the addition of new evidence, they will likely not engage in preparing new RCTs or other types of primary studies. 4 With the full appreciation of the notion that not all SRs need updating, we were very much interested particularly in Cochrane reviews that have been declared as stable (i.e., no longer needing an update) because of the conclusiveness of evidence. We hypothesized that a more detailed analysis of those reviews could reveal a consistent pattern of decision-making that goes into Cochrane's definition of "enough evidence."…”

“…This review will no longer be updated.”; “Conclusions unlikely to change with the addition of new studies.”; “This review will no longer be updated. The authors and editors are confident that further research will not change the conclusions.” 4 The next logical question is – what does it mean exactly that the evidence is conclusive; that is, how do we define a point where further research on a topic is no longer needed. Because, if Cochrane has come up with a definition of conclusive evidence in SRs, then this definition could be universally applied to non‐Cochrane SRs as well.…”

Quality of meta‐analyses in freshwater ecology: A systematic review

“…WoS and Scopus) used to find articles that provide data to a meta‐analysis, is far from being a mere bureaucratic step (Nakagawa et al, 2007; Gusenbauer & Haddaway, 2020). For example, researchers may avoid wasting time through the repetition of an inclusive meta‐analysis when search strings are reported or may detect the need for a new meta‐analysis when/if the one published was too restrictive (see Babić et al., 2020). However, our results parallel those of Koricheva and Gurevitch, (2014) by showing that only a few studies included full details of bibliographic searches.…”