Comparison of Pooled Risk Estimates for Adverse Effects from Different Observational Study Designs: Methodological Overview

Abstract

BackgroundA diverse range of study designs (e.g. case-control or cohort) are used in the evaluation of adverse effects. We aimed to ascertain whether the risk estimates from meta-analyses of case-control studies differ from that of other study designs.MethodsSearches were carried out in 10 databases in addition to reference checking, contacting experts, and handsearching key journals and conference proceedings. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from case-control studies could be directly compared with the pooled estimate for the same adverse effect arising from other types of observational studies.ResultsWe included 82 meta-analyses. Pooled estimates of harm from the different study designs had 95% confidence intervals that overlapped in 78/82 instances (95%). Of the 23 cases of discrepant findings (significant harm identified in meta-analysis of one type of study design, but not with the other study design), 16 (70%) stemmed from significantly elevated pooled estimates from case-control studies. There was associated evidence of funnel plot asymmetry consistent with higher risk estimates from case-control studies. On average, cohort or cross-sectional studies yielded pooled odds ratios 0.94 (95% CI 0.88–1.00) times lower than that from case-control studies.InterpretationEmpirical evidence from this overview indicates that meta-analysis of case-control studies tend to give slightly higher estimates of harm as compared to meta-analyses of other observational studies. However it is impossible to rule out potential confounding from differences in drug dose, duration and populations when comparing between study designs.

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

“…37,42 Concato et al 37 justified this exclusion based on previous evidence presented in Sacks et al,39 who reported that 79% of interventions tested were considered effective in trials with historical controls, whereas only 20% were considered effective in RCTs. Further empirical evidence of the potential for bias in studies using historical controls is also presented in Ioannidis et al, 34 Algra and Rothwell 40 and Golder et al, 41 who all found that there were fewer discrepancies between the results of RCTs and NRSs when studies with historical controls were excluded. Ioannidis et al 34 also found that results from prospective NRSs contained fewer discrepancies compared with effect estimates from randomised studies than did retrospective studies, either with current or historical controls.…”

“…[30][31][32][33][34][35][36][37][38][39][40][41][42][43] A summary of the methods and findings of each the 14 identified studies is presented in Appendix 2 (see Table 42). …”

“…In six of the studies, [33][34][35]37,38,41 data were sourced from published meta-analyses that included both RCTs and NRSs. Five other studies 30,32,39,40,42 took a different approach and searched for NRSs that compared treatment effects and then carried out a further search to locate relevant RCTs.…”

“…Of the 14 included studies, seven 30,35,37,38,[40][41][42] concluded that there were no systematic differences in either the size or the direction of effect estimates obtained from NRSs compared with those from RCTs. Five studies 31,32,34,39,43 concluded that effect estimates obtained from NRSs were systematically larger than those obtained from RCTs.…”

What Is the Plural of a ‘Yellow’ Anecdote?

“…Golder et al [8] showed that systematic reviews of randomized and observational data give more similar answers than expected.…”