Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

Hong, Yoon Duk; Jansen, Jeroen P.; Guerino, John; Berger, Marc L.; Crown, William H.; Goettsch, Wim; Mullins, C. Daniel; Willke, Richard J.; Orsini, Lucinda

doi:10.1186/s12916-021-02176-1

Cited by 9 publications

(12 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hong et al [ 103 ] conducted a meta-epidemiological study comparing 74 pairs of summary effect estimates from RCTs and observational studies in the field of pharmacology. On average, differences were small albeit with considerable between-study variability, which is in line with our findings.…”

Section: Discussionmentioning

confidence: 99%

Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

Bröckelmann

Balduzzi

Harms

et al. 2022

BMC Med

View full text Add to dashboard Cite

Background Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement. Methods Systematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements. Results One hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as “more or less identical”; 90 were rated as “similar but not identical” and 37 as only “broadly similar”. For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97–1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs. Conclusions On average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

Bröckelmann

Balduzzi

Harms

et al. 2022

BMC Med

View full text Add to dashboard Cite

show abstract

“…However, NDS ignores differences in study designs and cannot consider the potential bias of RWE studies [ 20 ]. Furthermore, compared with RCTs, the results of RWE studies often show a large effect because of some uncontrolled confounding bias factors [ 53 ], and their interval estimates are much smaller because the events and the sample size are usually much larger [ 21 ]. Therefore, with the inclusion of RWE studies in a rare events meta-analysis of RCTs using NDS, not only can the bias of RWE studies not be adjusted but it would also give a larger weight than that of RCTs.…”

Section: Discussionmentioning

confidence: 99%

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

Yao

Wang

Mei

et al. 2023

JCM

View full text Add to dashboard Cite

Background: Many rare events meta-analyses of randomized controlled trials (RCTs) have lower statistical power, and real-world evidence (RWE) is becoming widely recognized as a valuable source of evidence. The purpose of this study is to investigate methods for including RWE in a rare events meta-analysis of RCTs and the impact on the level of uncertainty around the estimates. Methods: Four methods for the inclusion of RWE in evidence synthesis were investigated by applying them to two previously published rare events meta-analyses: the naïve data synthesis (NDS), the design-adjusted synthesis (DAS), the use of RWE as prior information (RPI), and the three-level hierarchical models (THMs). We gauged the effect of the inclusion of RWE by varying the degree of confidence placed in RWE. Results: This study showed that the inclusion of RWE in a rare events meta-analysis of RCTs could increase the precision of the estimates, but this depended on the method of inclusion and the level of confidence placed in RWE. NDS cannot consider the bias of RWE, and its results may be misleading. DAS resulted in stable estimates for the two examples, regardless of whether we placed high- or low-level confidence in RWE. The results of the RPI approach were sensitive to the confidence level placed in RWE. The THM was effective in allowing for accommodating differences between study types, while it had a conservative result compared with other methods. Conclusion: The inclusion of RWE in a rare events meta-analysis of RCTs could increase the level of certainty of the estimates and enhance the decision-making process. DAS might be appropriate for inclusion of RWE in a rare event meta-analysis of RCTs, but further evaluation in different scenarios of empirical or simulation studies is still warranted.

show abstract

“…We will search PubMed to identify eligible systematic reviews. Our search strategy includes terms for systematic review, meta-analysis, RCTs and NRSIs (online supplemental appendix 1) and was informed by a previous related study 20 22 28…”

Section: Methods and Analysismentioning

confidence: 99%

“…Our sample size estimation is based on the proportion of agreement of treatment effects between NRSIs and RCTs (defined below). The sample size is calculated by the equation: n= zα2[p|1-p]δ2, where true0Zα=1.96, p denotes the proportion of the agreement of effects from NRSIs and RCTs and is equal to 80% based on two previous studies,22 23 δ is the tolerance of errors and is set as 5.5%. According to this estimation, we will require at least a total of 204 systematic reviews for our study.…”

Section: Methods and Analysismentioning

confidence: 99%

Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

Yao¹,

Wang²,

Busse³

et al. 2023

BMJ Open

View full text Add to dashboard Cite

IntroductionAlthough interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than RCTs. The objectives of this study are to: (1) explore how NRSIs can be integrated into a meta-analysis of RCTs; (2) assess concordance of the evidence from non-randomised and randomised trials and explore factors associated with agreement; and (3) investigate the impact on estimates of pooled bodies of evidence when NRSIs are included.Methods and analysisWe will conduct a systematic survey of 210 systematic reviews that include both RCTs and NRSIs, published from 2017 to 2022. We will randomly select reviews, stratified in a 1:1 ratio by Core vs non-Core clinical journals, as defined by the National Library of Medicine. Teams of paired reviewers will independently determine eligibility and abstract data using standardised, pilot-tested forms. The concordance of the evidence will be assessed by exploring agreement in the relative effect reported by NRSIs and RCT addressing the same clinical question, defined as similarity of the population, intervention/exposure, control and outcomes. We will conduct univariable and multivariable logistic regression analyses to examine the association of prespecified study characteristics with agreement in the estimates between NRSIs and RCTs. We will calculate the ratio of the relative effect estimate from NRSIs over that from RCTs, along with the corresponding 95% CI. We will use a bias-corrected meta-analysis model to investigate the influence on pooled estimates when NRSIs are included in the evidence synthesis.Ethics and disseminationEthics approval is not required. The findings of this study will be disseminated through peer-reviewed publications, conference presentations and condensed summaries for clinicians, health policymakers and guideline developers regarding the design, conduct, analysis, and interpretation of meta-analysis that integrate RCTs and NRSIs.

show abstract

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

Cited by 9 publications

References 57 publications

Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

Contact Info

Product

Resources

About