Synthesizing evidence from the earliest studies to support decision‐making: To what extent could the evidence be reliable?

Abstract

In evidence‐based practice, new topics generally only have a few studies available for synthesis. As a result, the evidence of such meta‐analyses raised substantial concerns. We investigated the robustness of the evidence from these earliest studies. Real‐world data from the Cochrane Database of Systematic Reviews (CDSR) were collected. We emulated meta‐analyses with the earliest 1 to 10 studies through cumulative meta‐analysis from eligible meta‐analyses. The magnitude and the direction of meta‐analyses with the earliest few studies were compared to the full meta‐analyses. From the CDSR, we identified 20,227 meta‐analyses of binary outcomes and 7683 meta‐analyses of continuous outcomes. Under the tolerable difference of 20% on the magnitude of the effects, the convergence proportion ranged from 24.24% (earliest 1 study) to 77.45% (earliest 10 studies) for meta‐analyses of few earliest studies with binary outcomes. For meta‐analyses of continuous outcomes, the convergence proportion ranged from 13.86% to 56.52%. In terms of the direction of the effects, even when only three studies were available at the earliest stage, the majority had the same direction as full meta‐analyses; Only 19% for binary outcomes and 12% for continuous outcomes changed the direction as further evidence accumulated. Synthesizing evidence from the earliest studies is feasible to support urgent decision‐making, and in most cases, the decisions would be reasonable. Considering the potential uncertainties, it is essential to evaluate the confidence of the evidence of these meta‐analyses and update the evidence when necessary.

Effects of Photobiomodulation on Pain and Return to Play of Injured Athletes: A Systematic Review and Meta-analysis

“…The findings indicated a positive effect on pain reduction for PBM vs. control groups, SMD 5 1.03, SE 5 0.22, 95% CI 5 [0.43-1.63], p 5 0.0089 (Figure 2). The 2 studies that evaluated time to return to play in number of days will be summarized in narrative because meta-analysis is not appropriate for 2 comparison groups (63). One study suggested that an aloe gel group returned the athlete to play quicker than the PBM group (3).…”

“…The 2 studies that evaluated time to return to play in number of days will be summarized in narrative because meta-analysis is not appropriate for 2 comparison groups (63). One study suggested that an aloe gel group returned the athlete to play quicker than the PBM group (3).…”

“…This systematic review only had 2 studies that assessed return to play, so a meta-analysis was not performed for this outcome. The magnitude and direction of effect size estimates from synthesizing data from less than 3 studies does not accurately forecast estimates when more studies are available and should not be used for decision making (63). The level of significance set for the study was p < 0.05.…”

The SMART Safety: An empirical dataset for evidence synthesis of adverse events

“…In addition, to be included in the dataset, systematic reviews should have at lead one meta-analysis that compares the risk of harms of one medication intervention to active drugs or inactive controls (e.g., placebo, waitlist), and the meta-analysis should have at least five randomized controlled trials, with the 2 by 2 table data were available for each trial. We limited meta-analyses should have five and more trials to reduce potential random error since one of our sub-projects was to investigate the statistical performance of current evidence synthesis methods [4] . We set requirement on the 2 by 2 table data since it allows us to identify any potential data extraction errors and take steps to solving the errors [5] .…”