High quality of evidence is uncommon in Cochrane systematic reviews in Anaesthesia, Critical Care and Emergency Medicine

Conway, Aaron; Conway, Zachary J.; Soalheira, Kathleen; Sutherland, Joanna

doi:10.1097/eja.0000000000000691

Cited by 16 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The previously conducted systematic review also considered secondary outcomes including the extent and duration of virus shedding and viremia, and the results and conclusions were similar to those presented here. For example, our review as well as that performed earlier by Marenzoni et al 54 revealed poor reporting quality of the selected studies. Their meta‐analysis using a random effects model failed to demonstrate that vaccines reduced the number of vaccinated horses with at least one clinical sign following virus challenge infection (pooled RR 0.97, 95% CI 0.86‐1.10, P = .62) with lower heterogeneity ( I 2 = 19%), which contrasts with the results reported here.…”

Section: Discussionmentioning

confidence: 53%

“…This observation is consistent with some other reviews of the human medical literature. 54,55 Factors that contribute to these lower-quality studies included risk of bias and imprecision. Similar downgrades were applied in the present study for concerns regarding precision, especially as a consequence of underpowered studies for detecting rare events including abortion and neurologic effects.…”

Section: Rating the Overall Quality Of Evidencementioning

confidence: 99%

See 1 more Smart Citation

Vaccination for the prevention of equine herpesvirus‐1 disease in domesticated horses: A systematic review and meta‐analysis

Osterrieder,

Dorman,

Burgess

et al. 2023

Veterinary Internal Medicne

View full text Add to dashboard Cite

BackgroundEquine herpes virus type 1 (EHV‐1) infection in horses is associated with respiratory and neurologic disease, abortion, and neonatal death.HypothesisVaccines decrease the occurrence of clinical disease in EHV‐1‐infected horses.MethodsA systematic review was performed searching multiple databases to identify relevant studies. Selection criteria were original peer‐reviewed research reports that investigated the in vivo use of vaccines for the prevention of disease caused by EHV‐1 in domesticated horses. Main outcomes of interest included pyrexia, abortion, neurologic disease, viremia, and nasal shedding. We evaluated risk of bias, conducted exploratory meta‐analyses of incidence data for the main outcomes, and performed a GRADE evaluation of the quality of evidence for each vaccine subtype.ResultsA total of 1018 unique studies were identified, of which 35 met the inclusion criteria. Experimental studies accounted for 31/35 studies, with the remainder being observational studies. Eight vaccine subclasses were identified including commercial (modified‐live, inactivated, mixed) and experimental (modified‐live, inactivated, deletion mutant, DNA, recombinant). Risk of bias was generally moderate, often because of underreporting of research methods, and sample sizes were small leading to imprecision in the estimate of the effect size. Several studies reported either no benefit or minimal vaccine efficacy for the primary outcomes of interest. Meta‐analyses revealed significant heterogeneity was present, and our confidence in the quality of evidence for most outcomes was low to moderate.Conclusions and Clinical ImportanceOur review indicates that commercial and experimental vaccines minimally reduce the incidence of clinical disease associated with EHV‐1 infection.

show abstract

Section: Discussionmentioning

confidence: 53%

Section: Rating the Overall Quality Of Evidencementioning

confidence: 99%

Vaccination for the prevention of equine herpesvirus‐1 disease in domesticated horses: A systematic review and meta‐analysis

Osterrieder,

Dorman,

Burgess

et al. 2023

Veterinary Internal Medicne

View full text Add to dashboard Cite

show abstract

“…If, for example, the trial authors exploit methodological flexibility to inflate treatment benefits or fail to power a study to detect harmful side effects, the systematic review that summarises this evidence will be unduly optimistic about treatment benefits and harms. Well-conducted systematic reviews can identify some methodological problems in trials: one review of Cochrane reviews found that high quality evidence was available for just 10% of the primary health outcomes examined (Conway et al, 2017 ). As Roberts et al ( 2015 ) point out, systematic reviews’ emphasis on including all relevant trials may be misplaced when so many trials have fundamental methodological flaws.…”

Section: Part Imentioning

confidence: 99%

Broad Medical Uncertainty and the ethical obligation for openness

2022

View full text Add to dashboard Cite

This paper argues that there exists a collective epistemic state of ‘Broad Medical Uncertainty’ (BMU) regarding the effectiveness of many medical interventions. We outline the features of BMU, and describe some of the main contributing factors. These include flaws in medical research methodologies, bias in publication practices, financial and other conflicts of interest, and features of how evidence is translated into practice. These result in a significant degree of uncertainty regarding the effectiveness of many medical treatments and unduly optimistic beliefs about the benefit/harm profiles of such treatments. We argue for an ethical presumption in favour of openness regarding BMU as part of a ‘Corrective Response’. We then consider some objections to this position (the ‘Anti-Corrective Response’), including concerns that public honesty about flaws in medical research could undermine trust in healthcare institutions. We suggest that, as it stands, the Anti-Corrective Response is unconvincing.

show abstract

“…To perform an external validation of our extraction procedures used in data set construction, we used human annotations from the small-scale study by Wilczynski et al [ 39 ], in which various elements were manually extracted from SoFs. Although a larger study was carried out by Conway et al [ 40 ], we were unable to obtain their data. The data set in the study by Wilczynski et al [ 39 ] contains 103 instances (reviews) on anesthesia, critical care, and emergency medicine from the CDSR.…”

Section: Methodsmentioning

confidence: 88%

Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study

Šuster¹,

Baldwin²,

Lau³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

Background Assessment of the quality of medical evidence available on the web is a critical step in the preparation of systematic reviews. Existing tools that automate parts of this task validate the quality of individual studies but not of entire bodies of evidence and focus on a restricted set of quality criteria. Objective We proposed a quality assessment task that provides an overall quality rating for each body of evidence (BoE), as well as finer-grained justification for different quality criteria according to the Grading of Recommendation, Assessment, Development, and Evaluation formalization framework. For this purpose, we constructed a new data set and developed a machine learning baseline system (EvidenceGRADEr). Methods We algorithmically extracted quality-related data from all summaries of findings found in the Cochrane Database of Systematic Reviews. Each BoE was defined by a set of population, intervention, comparison, and outcome criteria and assigned a quality grade (high, moderate, low, or very low) together with quality criteria (justification) that influenced that decision. Different statistical data, metadata about the review, and parts of the review text were extracted as support for grading each BoE. After pruning the resulting data set with various quality checks, we used it to train several neural-model variants. The predictions were compared against the labels originally assigned by the authors of the systematic reviews. Results Our quality assessment data set, Cochrane Database of Systematic Reviews Quality of Evidence, contains 13,440 instances, or BoEs labeled for quality, originating from 2252 systematic reviews published on the internet from 2002 to 2020. On the basis of a 10-fold cross-validation, the best neural binary classifiers for quality criteria detected risk of bias at 0.78 F1 (P=.68; R=0.92) and imprecision at 0.75 F1 (P=.66; R=0.86), while the performance on inconsistency, indirectness, and publication bias criteria was lower (F1 in the range of 0.3-0.4). The prediction of the overall quality grade into 1 of the 4 levels resulted in 0.5 F1. When casting the task as a binary problem by merging the Grading of Recommendation, Assessment, Development, and Evaluation classes (high+moderate vs low+very low-quality evidence), we attained 0.74 F1. We also found that the results varied depending on the supporting information that is provided as an input to the models. Conclusions Different factors affect the quality of evidence in the context of systematic reviews of medical evidence. Some of these (risk of bias and imprecision) can be automated with reasonable accuracy. Other quality dimensions such as indirectness, inconsistency, and publication bias prove more challenging for machine learning, largely because they are much rarer. This technology could substantially reduce reviewer workload in the future and expedite quality assessment as part of evidence synthesis.

show abstract

High quality of evidence is uncommon in Cochrane systematic reviews in Anaesthesia, Critical Care and Emergency Medicine

Cited by 16 publications

References 15 publications

Vaccination for the prevention of equine herpesvirus‐1 disease in domesticated horses: A systematic review and meta‐analysis

Vaccination for the prevention of equine herpesvirus‐1 disease in domesticated horses: A systematic review and meta‐analysis

Broad Medical Uncertainty and the ethical obligation for openness

Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study

Contact Info

Product

Resources

About