Empirical evaluation of the methods used in systematic reviews including observational studies and randomized trials

Cheurfa, Chérifa; Tsokani, Sofia; Kontouli, Katerina-Maria; Boutron, Isabelle; Chaimani, Anna

doi:10.1016/j.jclinepi.2023.02.008

Cited by 3 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because we limited our scope to guidelines and tools published after 2005 to capture the period after publication of key overadjustment papers, we did not include some key guidelines (e.g. Meta-Analysis of Observational Studies in Epidemiology: MOOSE 19 ) and tools that have been very commonly used in the literature 7 , 9 , 10 (e.g. the Newcastle Ottawa Scales 20 and Downs and Black tool 21 )—regardless, none of these considered overadjustment bias.…”

Section: Discussionmentioning

confidence: 99%

“…ROBINS-I 15 , ROBINS-E 16 ). More broadly, as identified elsewhere, 10 there is a need to ensure that guidelines are tailored to reviews that include observational studies, including addressing the types of bias (notably overadjustment bias and confounding bias) that are particularly relevant to observational studies compared with RCTs. Given that uptake and adherence to guidelines and tools are suboptimal, 5 , 6 multifaceted actions are needed to address this across all stages of research and from all relevant stakeholders (including researchers, editors, peer reviewers, publishers, ethics committees, universities and funders).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Consideration of overadjustment bias in guidelines and tools for systematic reviews and meta-analyses of observational studies is long overdue

van Zwieten,

Blyth,

Wong

et al. 2023

International Journal of Epidemiology

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Consideration of overadjustment bias in guidelines and tools for systematic reviews and meta-analyses of observational studies is long overdue

van Zwieten,

Blyth,

Wong

et al. 2023

International Journal of Epidemiology

View full text Add to dashboard Cite

“…The search strategy and selection process for eligible systematic reviews have been previously described in detail [ 24 ]. Briefly, we searched Medline via PubMed to identify systematic reviews that included RCTs and OSs evaluating the effect of healthcare interventions, published between January 2015 and December 2019 in general and internal medicine or public health journals with an impact factor ≥ 2.5 or in the top five specialty medical journals.…”

Section: Methodsmentioning

confidence: 99%

“…We have previously explored the approaches that review authors used at the systematic review stage (data extraction, risk of bias assessment, etc.) to reassure that the specific challenges and potential biases introduced by the inclusion of observational studies will be adequately accommodated in their conclusions [ 24 ]. We found that published systematic reviews including RCTs and OSs were often lacking proper reporting and methodology for OSs; for instance, few reviews only reported registration of a protocol and adjusted estimates were rarely extracted.…”

Section: Introductionmentioning

confidence: 99%

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses

Cheurfa,

Tsokani,

Kontouli

et al. 2024

Syst Rev

Self Cite

View full text Add to dashboard Cite

Background This study examined the synthesis methods used in meta-analyses pooling data from observational studies (OSs) and randomised controlled trials (RCTs) from various medical disciplines. Methods We searched Medline via PubMed to identify reports of systematic reviews of interventions, including and pooling data from RCTs and OSs published in 110 high-impact factor general and specialised journals between 2015 and 2019. Screening and data extraction were performed in duplicate. To describe the synthesis methods used in the meta-analyses, we considered the first meta-analysis presented in each article. Results Overall, 132 reports were identified with a median number of included studies of 14 [9–26]. The median number of OSs was 6.5 [3–12] and that of RCTs was 3 [1–6]. The effect estimates recorded from OSs (i.e., adjusted or unadjusted) were not specified in 82% (n = 108) of the meta-analyses. An inverse-variance common-effect model was used in 2% (n = 3) of the meta-analyses, a random-effects model was used in 55% (n = 73), and both models were used in 40% (n = 53). A Poisson regression model was used in 1 meta-analysis, and 2 meta-analyses did not report the model they used. The mean total weight of OSs in the studied meta-analyses was 57.3% (standard deviation, ± 30.3%). Only 44 (33%) meta-analyses reported results stratified by study design. Of them, the results between OSs and RCTs had a consistent direction of effect in 70% (n = 31). Study design was explored as a potential source of heterogeneity in 79% of the meta-analyses, and confounding factors were investigated in only 10% (n = 13). Publication bias was assessed in 70% (n = 92) of the meta-analyses. Tau-square was reported in 32 meta-analyses with a median of 0.07 [0–0.30]. Conclusion The inclusion of OSs in a meta-analysis on interventions could provide useful information. However, considerations of several methodological and conceptual aspects of OSs, that are required to avoid misleading findings, were often absent or insufficiently reported in our sample.

show abstract

Integration of non-randomized studies with randomized controlled trials in meta-analyses of clinical studies: a meta-epidemiological study on effect estimation of interventions

Mei,

Yao,

Wang

et al. 2024

BMC Med

View full text Add to dashboard Cite

Backgrounds Syntheses of non-randomized studies of interventions (NRSIs) and randomized controlled trials (RCTs) are increasingly used in decision-making. This study aimed to summarize when NRSIs are included in evidence syntheses of RCTs, with a particular focus on the methodological issues associated with combining NRSIs and RCTs. Methods We searched PubMed to identify clinical systematic reviews published between 9 December 2017 and 9 December 2022, randomly sampling reviews in a 1:1 ratio of Core and non-Core clinical journals. We included systematic reviews with RCTs and NRSIs for the same clinical question. Clinical scenarios for considering the inclusion of NRSIs in eligible studies were classified. We extracted the methodological characteristics of the included studies, assessed the concordance of estimates between RCTs and NRSIs, calculated the ratio of the relative effect estimate from NRSIs to that from RCTs, and evaluated the impact on the estimates of pooled estimates when NRSIs are included. Results Two hundred twenty systematic reviews were included in the analysis. The clinical scenarios for including NRSIs were grouped into four main justifications: adverse outcomes ( n = 140, 63.6%), long-term outcomes ( n = 36, 16.4%), the applicability of RCT results to broader populations ( n = 11, 5.0%), and other ( n = 33, 15.0%). When conducting a meta-analysis, none of these reviews assessed the compatibility of the different types of evidence prior, 203 (92.3%) combined estimates from RCTs and NRSIs in the same meta-analysis. Of the 203 studies, 169 (76.8%) used crude estimates of NRSIs, and 28 (13.8%) combined RCTs and multiple types of NRSIs. Seventy-seven studies (35.5%) showed “qualitative disagree” between estimates from RCTs and NRSIs, and 101 studies (46.5%) found “important difference”. The integration of NRSIs changed the qualitative direction of estimates from RCTs in 72 out of 200 studies (36.0%). Conclusions Systematic reviews typically include NRSIs in the context of assessing adverse or long-term outcomes. The inclusion of NRSIs in a meta-analysis of RCTs has a substantial impact on effect estimates, but discrepancies between RCTs and NRSIs are often ignored. Our proposed recommendations will help researchers to consider carefully when and how to synthesis evidence from RCTs and NRSIs. Supplementary Information The online version contains supplementary material available at 10.1186/s12916-024-03778-1.

show abstract

Empirical evaluation of the methods used in systematic reviews including observational studies and randomized trials

Cited by 3 publications

References 36 publications

Consideration of overadjustment bias in guidelines and tools for systematic reviews and meta-analyses of observational studies is long overdue

Consideration of overadjustment bias in guidelines and tools for systematic reviews and meta-analyses of observational studies is long overdue

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses

Integration of non-randomized studies with randomized controlled trials in meta-analyses of clinical studies: a meta-epidemiological study on effect estimation of interventions

Contact Info

Product

Resources

About