ObjectivesTo investigate the validity of data extraction in systematic reviews of adverse events, the effect of data extraction errors on the results, and to develop a classification framework for data extraction errors to support further methodological research.DesignReproducibility study.Data sourcesPubMed was searched for eligible systematic reviews published between 1 January 2015 and 1 January 2020. Metadata from the randomised controlled trials were extracted from the systematic reviews by four authors. The original data sources (eg, full text and ClinicalTrials.gov) were then referred to by the same authors to reproduce the data used in these meta-analyses.Eligibility criteria for selecting studiesSystematic reviews were included when based on randomised controlled trials for healthcare interventions that reported safety as the exclusive outcome, with at least one pair meta-analysis that included five or more randomised controlled trials and with a 2×2 table of data for event counts and sample sizes in intervention and control arms available for each trial in the meta-analysis.Main outcome measuresThe primary outcome was data extraction errors summarised at three levels: study level, meta-analysis level, and systematic review level. The potential effect of such errors on the results was further investigated.Results201 systematic reviews and 829 pairwise meta-analyses involving 10 386 randomised controlled trials were included. Data extraction could not be reproduced in 1762 (17.0%) of 10 386 trials. In 554 (66.8%) of 829 meta-analyses, at least one randomised controlled trial had data extraction errors; 171 (85.1%) of 201 systematic reviews had at least one meta-analysis with data extraction errors. The most common types of data extraction errors were numerical errors (49.2%, 867/1762) and ambiguous errors (29.9%, 526/1762), mainly caused by ambiguous definitions of the outcomes. These categories were followed by three others: zero assumption errors, misidentification, and mismatching errors. The impact of these errors were analysed on 288 meta-analyses. Data extraction errors led to 10 (3.5%) of 288 meta-analyses changing the direction of the effect and 19 (6.6%) of 288 meta-analyses changing the significance of the P value. Meta-analyses that had two or more different types of errors were more susceptible to these changes than those with only one type of error (for moderate changes, 11 (28.2%) of 39 v 26 (10.4%) 249, P=0.002; for large changes, 5 (12.8%) of 39 v 8 (3.2%) of 249, P=0.01).ConclusionSystematic reviews of adverse events potentially have serious issues in terms of the reproducibility of the data extraction, and these errors can mislead the conclusions. Implementation guidelines are urgently required to help authors of future systematic reviews improve the validity of data extraction.

To evaluate the efficacy and safety of human amniotic membrane (HAM) allograft in treating chronic diabetic foot ulcers (DFUs), a comprehensive search of randomised controlled trials in MEDLINE, EMBASE, PubMed, CENTRAL and Web of Science was conducted to December 7, 2019. Two reviewers independently screened the studies, extracted data, and evaluated the quality of studies. The primary outcome was the proportion of complete healing. The secondary outcomes were mean time to complete healing and adverse events. Statistical analyses were performed using RevMan 5.3. We identified 257 articles, of which 7 articles (465 participants) were included in the meta‐analysis. The proportion of complete wound healing in HAM plus standard of care (SOC) group was 3.88 times as high as that in SOC alone (RR: 3.88 [95% CI: 2.34, 6.44]) at 6 weeks, and 2.01 times at 12 weeks (RR: 2.01 [95%CI: 1.45, 2.77]). The intervention group had a significantly shorter time to complete healing (MD: −30.33 days, [95% CI: −37.95, −22.72]). The number needed to treat within 6 weeks was 2.3 ([95% CI: 1.8, 3.1]). No significant difference was shown in adverse events. Results were consistent in a sensitivity analysis. Hence, HAM plus SOC is effective and safe in treating chronic DFUs.

Background: Many treatments are currently available for amblyopic patients; although, the comparative efficacy of these therapies is unclear. We conducted a systematic review and network meta-analysis (NMA) to establish the relative efficacy of these treatments for amblyopia. Methods: Electronic databases (MEDLINE, EMBASE, Cochrane Library) were systematically searched from inception to Sep. 2019. Only Randomized clinical trials comparing any two or three of the following treatments were included: refractive correction (spectacles alone), patching of 2 h per day (patch 2H), patch 6H, patch 12H, patch 2H + near activities (N), patch 2H + distant activities (D), atropine (Atr) daily, Atr weekly, Atr weekly + plano lens over the sound eye (Plano), optical penalization and binocular therapy. The reviewers independently extracted the data according to the PRISMA guidelines; assessed study quality by Cochrane risk-of-bias tool for randomized trials. The primary outcome measure was the change in best-corrected visual acuity (BCVA) expressed as log MAR lines. Direct comparisons and a Bayesian meta-analysis were performed to synthesize data. Results: Twenty-three studies with 3279 patients were included. In the network meta-analysis, optical penalization was the least effective of all the treatments for the change of visual acuity, spectacles (mean difference [MD]

To evaluate the efficacy and safety of recombinant human epidermal growth factor (rhEGF) in treating diabetic foot ulcers (DFUs), we conducted both database searches (PubMed, MEDLINE, EMBASE, CENTRAL, and Web of Science) and reference searches for randomised controlled trials from the inception of databases to 30 January 2020. Two reviewers independently scrutinised the trials, extracted data, and assessed the quality of trials. The primary outcome was the proportion of complete healing. The secondary outcomes were mean time to complete healing and adverse events. A subgroup analysis was performed by different administration routes. Statistical analyses were performed in RevMan 5.3. The time to complete healing Kaplan‐Meier curves was pooled in the R software. Of the 156 citations, 9 trials (720 participants) met eligibility criteria and were included. The rhEGF achieved a higher complete healing rate than placebo (OR: 2.79, [95% CI: 1.99, 3.99]). The rhEGF also significantly shorten complete healing time (MD: −14.10 days, [95% CI: −18.03, −10.16]). Subgroup analysis showed that topical application was superior to intralesional injection, but that may be because of different ulcer severity they included. No significant difference was shown in adverse events. Results were coherent with sensitivity analyses. Therefore, rhEGF is an effective and safe treatment for DFUs.

In evidence‐based practice, new topics generally only have a few studies available for synthesis. As a result, the evidence of such meta‐analyses raised substantial concerns. We investigated the robustness of the evidence from these earliest studies. Real‐world data from the Cochrane Database of Systematic Reviews (CDSR) were collected. We emulated meta‐analyses with the earliest 1 to 10 studies through cumulative meta‐analysis from eligible meta‐analyses. The magnitude and the direction of meta‐analyses with the earliest few studies were compared to the full meta‐analyses. From the CDSR, we identified 20,227 meta‐analyses of binary outcomes and 7683 meta‐analyses of continuous outcomes. Under the tolerable difference of 20% on the magnitude of the effects, the convergence proportion ranged from 24.24% (earliest 1 study) to 77.45% (earliest 10 studies) for meta‐analyses of few earliest studies with binary outcomes. For meta‐analyses of continuous outcomes, the convergence proportion ranged from 13.86% to 56.52%. In terms of the direction of the effects, even when only three studies were available at the earliest stage, the majority had the same direction as full meta‐analyses; Only 19% for binary outcomes and 12% for continuous outcomes changed the direction as further evidence accumulated. Synthesizing evidence from the earliest studies is feasible to support urgent decision‐making, and in most cases, the decisions would be reasonable. Considering the potential uncertainties, it is essential to evaluate the confidence of the evidence of these meta‐analyses and update the evidence when necessary.