Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Xu, Chang; Yu, Tianqi; Furuya‐Kanamori, Luis; Lin, Lifeng; Zorzela, Liliane; Zhou, Xiaoqin; Dai, Hanming; Loke, Yoon K.; Vohra, Sunita

doi:10.1136/bmj-2021-069155

Cited by 33 publications

(43 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The studies used for data extraction in the two groups will be identified before the start of the trial, based on our previous well-established database of meta-analyses of adverse events (binary outcomes) [7]. The database covers 201 systematic reviews of randomized controlled trials with 829 meta-analyses with pharmaceutical or non-pharmaceutical interventions, and all the meta-analytical data in the database have been carefully checked for their validity.…”

Section: Methodsmentioning

confidence: 99%

“…Systematic review and meta-analysis has been one of the most important sources of evidence and thus the validity of such evidence directly determines the reliability and quality of healthcare administration [2]. Unfortunately, in real-world, the evidence generated from systematic reviews and meta-analyses is far from valid or trust-worthy due to a multiple reasons, where one of which would be errors during data extraction -as recorded in previous studies, as much as 85% of the systematic reviews faces serious issue in data reproducibility [3][4][5][6][7].…”

Section: Introductionmentioning

confidence: 99%

“…While evidence from two randomized trials suggest some benefits of duplicate data extraction for improving data reproducibility [10, 11]; our empirical investigation suggest that the error rates may differ by the type of interventions. We found that meta-analyses assessing pharmaceutical interventions had almost twice the error rate than investigating non-pharmaceutical interventions [6,7]. The above findings sparked our interest in why the error rates differ in meta-analyses with pharmaceutical and non-pharmaceutical interventions, and whether duplicate data extraction could ‘trade-off’ such preference on the occurrence of errors?…”

Section: Introductionmentioning

confidence: 99%

“…Unfortunately, in real-world, the evidence from systematic reviews and meta-analyses were far from valid or trust-worthy due to a package of reasons, where one of which would be the frequently occurred errors during data extraction -as recorded in previous literatures, as much as 85% of the systematic reviews faces serious issue in data reproducibility [3][4][5][6][7].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Data extraction error in pharmaceutical versus non-pharmaceutical interventions for evidence synthesis: Study protocol for a crossover trial

Zhu

Ren

Doi

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundData extraction is the foundation for a trusted systematic review; it is often time- and labor-consuming and prone to errors. Whilst data extraction errors frequently occur in literatures, an interesting phenomenon was observed that such erroneous risks tend to be higher in trials of pharmaceutical interventions compared to non-pharmaceutical one. This phenomenon has not been verified by high-quality evidence; The elucidation of which would have implications for guidelines, practice and policy.Methods and analysesWe prepare a 1:1 designed, randomized, multicenter, investigator-blinded, cross-over trial to elucidate the potential variants on the error rates of meta-analyses with pharmaceutical against non-pharmaceutical interventions. Eligible participants that at the 2nd year of their post-graduate program or above (e.g., doctoral program) will be recruited and randomly assigned to one of the two groups to complete pre-defined data extraction tasks: 1) group A contains 10 randomized controlled trials (RCTs) of pharmaceutical interventions; 2) group B contains 10 RCTs of non-pharmaceutical interventions. Participants who would finish the data extraction will be further assigned to the alternative group for another round of data extraction, after at least 30 mins break. Finally, those participants assigned to A or B group will be further 1:1 randomly matched based on a random-sequenced number, for the double-checking process on the extracted data. The primary outcome will be the error rates on data extraction of the pharmaceutical intervention group and the non-pharmaceutical group, before the double-checking process, in terms of the cell level, study level, and participant level. The secondary outcome will be the error rates on data extraction of the pharmaceutical intervention group and the non-pharmaceutical group, after the double-checking process, again, in terms of the cell level, study level, and participant level. The generalized linear mixed effects model (based on the above three levels) will be used to estimate the potential differences on the error rates, with a link function of log that refers to binomial distribution. Subgroup analyses are planned by the following factors: the experience of individuals on systematic reviews, time used for the data extraction.Ethics and disseminationThis study has been approved by the institutional review board of Anhui Medical University. We plan to present our findings at international scientific meetings, and publish our findings in a high-quality peer-reviewed academic journal.Trial registrationChinese Clinical Trial Register Center (Identifier: ChiCTR2200062206).Strengths and limitations of the studyThis will be the first trial to test the error rates of data extraction in the sight of intervention type that characterized by pharmaceutical intervention or non-pharmaceutical intervention.This will be the third randomized trial on strategy of data extraction in the world and the first in Asian-Pacific area.The use of a cross-over design provides a valid way to eliminate the potential impact of the heterogeneous contexts of the studies and thus is expected to provide less biased evidence to support a better evidence synthesis practice.The restrict the participants to 2nd year post-graduate students or above to ensure the feasibility of the trial, which is no doubt impact the representativeness of the samples.A group of useful strategies should be taken to minimize the impact of the possible sharing of completed extraction table among the participants.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Data extraction error in pharmaceutical versus non-pharmaceutical interventions for evidence synthesis: Study protocol for a crossover trial

Zhu

Ren

Doi

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The “no-humbug” expression used by Dr. Warren was not a surprising choice. We must think about innovations consolidated by evidence-based care and rigorous scientific studies to avoid dire consequences that can put patients at risk (Slayton et al 2018; Howick et al 2022; Xu et al 2022), such as the failing protoplast/Teflon implants in the 1980s and 1990s that highly affected thousands of lives of patients with temporomandibular disorders (Lee et al 2018). There is great potential to be more integrated than ever with patients and their other care providers via devices and combining medical and dental electronic health records.…”

Section: Introductionmentioning

confidence: 99%

The Forefront of Dentistry—Promising Tech-Innovations and New Treatments

DaSilva

Robinson

Shi³

et al. 2022

JDR Clinical & Translational Research

View full text Add to dashboard Cite

Change in our world is happening quickly. For dentistry, this is no exception, and now is the time to foster new opportunities. Advances in the foundational science of dental, oral, and craniofacial health are expanding and enabling personalized care built on strong discoveries in the fields of microbiology, immunology, and neuroscience. Digital communication through electronic health records, patient portals, and teledentistry is emerging as vital new armamentarium for contemporary dental practice. Wearables linked with smartphone apps are trending to monitor pain levels, postoperative symptoms, wound healing, and home oral health habits. Oral fluid diagnostics are rapidly surfacing for use in diagnostics, drug testing, and hormone detection. Manipulation of the oral microbiome is under investigation for caries management and for potential impact on systemic conditions. Resolving mediators of inflammation are promising for addressing the chronic inflammation that plagues periodontal patients. New imaging modalities paired with digital approaches and 3-dimensional printing are formulating disruptions in the technical aspects of dentistry. Augmented reality and virtual reality are providing great potential as teaching tools for dental students and bear promise for lifelong learning for providers. Optimization of information technology and the incorporation of data science partnered with artificial intelligence and machine learning to improve patient care in a learning health care system approach will benefit large numbers of patients. These are exciting times for our profession, yet we need to navigate these paths carefully. Technology needs to significantly and positively change the experience of those involved (patients, dentists, staff, payors). New technologies need to be backed with solid research and implementation science. Automation should focus on improving care with attention to expanding care to the underserved and not lose the critical human connection that our patients need. There is great potential in the emerging innovations yet also great responsibility to ensure they are evidence based on rigorous science and deployable with equity and sensitivity. Knowledge Transfer Statement: This article discusses innovations in technology and treatments that have enormous potential to revolutionize our dental care, including novel concepts in electronic health records, communication between dentists and patients, biologics around diagnosis and treatment, digital dentistry, and, finally, the real-time optimization of information technology. The early implementation and validation of these innovations can drive down their costs and provide better dental and medical services to all members of our society.

show abstract

Synthesizing evidence from the earliest studies to support decision‐making: To what extent could the evidence be reliable?

Lin

Furuya‐Kanamori

et al. 2022

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

In evidence‐based practice, new topics generally only have a few studies available for synthesis. As a result, the evidence of such meta‐analyses raised substantial concerns. We investigated the robustness of the evidence from these earliest studies. Real‐world data from the Cochrane Database of Systematic Reviews (CDSR) were collected. We emulated meta‐analyses with the earliest 1 to 10 studies through cumulative meta‐analysis from eligible meta‐analyses. The magnitude and the direction of meta‐analyses with the earliest few studies were compared to the full meta‐analyses. From the CDSR, we identified 20,227 meta‐analyses of binary outcomes and 7683 meta‐analyses of continuous outcomes. Under the tolerable difference of 20% on the magnitude of the effects, the convergence proportion ranged from 24.24% (earliest 1 study) to 77.45% (earliest 10 studies) for meta‐analyses of few earliest studies with binary outcomes. For meta‐analyses of continuous outcomes, the convergence proportion ranged from 13.86% to 56.52%. In terms of the direction of the effects, even when only three studies were available at the earliest stage, the majority had the same direction as full meta‐analyses; Only 19% for binary outcomes and 12% for continuous outcomes changed the direction as further evidence accumulated. Synthesizing evidence from the earliest studies is feasible to support urgent decision‐making, and in most cases, the decisions would be reasonable. Considering the potential uncertainties, it is essential to evaluate the confidence of the evidence of these meta‐analyses and update the evidence when necessary.

show abstract

Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Cited by 33 publications

References 38 publications

Data extraction error in pharmaceutical versus non-pharmaceutical interventions for evidence synthesis: Study protocol for a crossover trial

Data extraction error in pharmaceutical versus non-pharmaceutical interventions for evidence synthesis: Study protocol for a crossover trial

The Forefront of Dentistry—Promising Tech-Innovations and New Treatments

Synthesizing evidence from the earliest studies to support decision‐making: To what extent could the evidence be reliable?

Contact Info

Product

Resources

About