When is the evidence conclusive? Analysis of systematic reviews for which Cochrane declared that conclusions will not change with further studies

Abstract

Some Cochrane reviews were declared stable or closed, that is, not in need of updating. For some of them, it has been declared that conclusions will not (or it is unlikely they will) change with further studies. We explored whether there is a discernable decision-making pattern for decisions about the conclusiveness and stabilization of these reviews. We analyzed Cochrane reviews published until April 2020 labeled as stable or closed. We extracted the rationale leading to the decision declaring that the conclusion is not expected to change with further studies. Furthermore, we assessed whether the reviews used GRADE analysis. We extracted data from summary of findings (SoF) tables on the direction of effect, statistical significance, and I 2 values for the first and primary outcomes in SoFs, conclusions in the abstract and review, and implications for practice and future research. We included 40 stable/closed Cochrane reviews. Rationales for their stabilization did not enable any insight into the Cochrane's decision-making algorithm for considering the evidence as conclusive. Among 191 outcomes presented in the SoFs, 70% were rated with either low or very low certainty evidence. None of the reviews mentioned in the text that the review should be stabilized or closed, or that there is sufficient evidence on the subject. Reasons for stabilizing/closing Cochrane reviews were unclear, and we could not discern any pattern of "conclusive review" traits. Definition of systematic review conclusiveness is still lacking, which may contribute to research waste.

Definition, harms, and prevention of redundant systematic reviews

“…That means that even if the latest search finds new relevant evidence, it can be considered that the new evidence is not sufficiently substantive to warrant an update. Even though currently it is unclear when is a systematic review conclusive and should not be updated any more [ 24 ], the authors should be very cautious when deciding whether an update of a systematic review is warranted.…”

“…Currently, we do not have a definition of conclusive research, i.e., a method to declare that we have had enough/sufficient evidence, and that new research on a topic is no longer needed [23,24,42]. If we could define that, we could declare that some research topics need to be closed, and that it is not necessary to conduct further research on this topic, primary or secondary.…”

Evaluation of ‘implications for research’ sections in systematic reviews of interventions in advanced cancer patients – a meta-research study

“…Another eld of further research could be the identi cation of methodological and clinical stop criteria for further studies and the de nition of conclusiveness in systematic reviews with or without meta-analysis [19,20,26]. This could either be a part of the above-mentioned IfR guideline or result in a separate guideline due to complexity of the question.…”

Proposed triggers for retiring a living systematic review

“…Such triggers for retirement from the living mode can be derived from relevant literature about when to start an LSR and from literature about updating non-LSRs 2 4 11–14. The Cochrane Collaboration suggests that an LSR may transition out of a living mode when the criteria to initiate LSR are no longer met 4.…”

“…A survey of 545 closed or stable non-living Cochrane reviews demonstrated that common rationales provided by authors to deem a topic stable related to not having new studies published on the topic, conclusiveness of the evidence or the fact that the intervention is no longer in general use 11. The justification was deemed insufficient in 49% of the reviews,11 and reanalysis of a sample of 40 reviews demonstrated that there was no clear algorithm or consistent approach to make such decision 12. In this proposal, we aim to identify potential situations in which LSRs can be retired from the living mode and illustrate these situations through a published example of an LSR that has published its last update and was retired from the living mode 9…”