Challenges in meta-analyses with observational studies

Metelli, Silvia; Chaimani, Anna

doi:10.1136/ebmental-2019-300129

Cited by 191 publications

(184 citation statements)

References 22 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…However, in practice, it is difficult to ensure the quality of original studies. Therefore, the potential biases of original studies should be carefully evaluated [ 12 ], and the consequent clinical or methodological heterogeneity among original studies should be controlled, either by statistical approaches such as random-effects model or meta-regression, or by subgroup analysis to pool all “combinable” results together [ 34 , 35 ]. Otherwise, if the methodologies of original studies are far from each other, qualitative systemic reviews rather than quantitative pooled analyses are recommended [ 9 , 36 ].…”

Section: Discussionmentioning

confidence: 99%

Can statistical adjustment guided by causal inference improve the accuracy of effect estimation? A simulation and empirical research based on meta-analyses of case–control studies

Yan

Liu

Peng

et al. 2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background Statistical adjustment is often considered to control confounding bias in observational studies, especially case–control studies. However, different adjustment strategies may affect the estimation of odds ratios (ORs), and in turn affect the results of their pooled analyses. Our study is aimed to investigate how to deal with the statistical adjustment in case–control studies to improve the validity of meta-analyses. Methods Three types of adjustment strategies were evaluated including insufficient adjustment (not all preset confounders were adjusted), full adjustment (all confounders were adjusted under the guidance of causal inference), and improper adjustment (covariates other than confounders were adjusted). We carried out a series of Monte Carlo simulation experiments based on predesigned scenarios, and assessed the accuracy of effect estimations from meta-analyses of case–control studies by combining ORs calculated according to different adjustment strategies. Then we used the data from an empirical review to illustrate the replicability of the simulation results. Results For all scenarios with different strength of causal relations, combining ORs that were comprehensively adjusted for confounders would get the most precise effect estimation. By contrast, combining ORs that were not sufficiently adjusted for confounders or improperly adjusted for mediators or colliders would easily introduce bias in causal interpretation, especially when the true effect of exposure on outcome was weak or none. The findings of the simulation experiments were further verified by the empirical research. Conclusions Statistical adjustment guided by causal inference are recommended for effect estimation. Therefore, when conducting meta-analyses of case–control studies, the causal relationship formulated by exposure, outcome, and covariates should be firstly understood through a directed acyclic graph, and then reasonable original ORs could be extracted and combined by suitable methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Can statistical adjustment guided by causal inference improve the accuracy of effect estimation? A simulation and empirical research based on meta-analyses of case–control studies

Yan

Liu

Peng

et al. 2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

show abstract

“…What may be more useful is to assess whether it is likely that studies included are “functionally identical” [ 29 ] as assumed under a fixed-effect model. Widespread differences in participant characteristics, intervention designs, settings and outcomes, make the absence of heterogeneity unlikely [ 28 , 33 , 34 ]. Public health interventions are even less likely to be homogenous; they often have interacting components targeting multiple groups, accommodate flexible delivery, and are embedded within complex systems [ 35 ].…”

Section: Main Textmentioning

confidence: 99%

“…Firstly, meta-analyses produce “observational” results even if randomized controlled-trials (RCTs) are included as random allocation is not preserved [ 43 ]. Observational studies, where assignment to comparison groups is not random, are considered to be at even higher risk for selection bias and confounding than RCTs [ 34 ]. While a random effects model is more suitable for MWH studies, the pooled estimates presented here may still be compromised by bias and confounding inherent to observational designs.…”

Section: Limitationsmentioning

confidence: 99%

“…Ideally, however, additional individual studies using robust designs are required for results from meta-analyses to be more informative. RCTs are generally accepted as providing the highest quality evidence [ 34 ] if well designed, conducted and reported. Where it is not feasible or ethical to conduct trials, longitudinal studies with careful participant selection, adequate confounder information, sufficient follow-up levels that analyse data suitably may be acceptable alternatives.…”

Section: Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

The effect of maternity waiting homes on perinatal mortality is inconclusive: a critical appraisal of existing evidence from Sub-Saharan Africa

et al. 2021

View full text Add to dashboard Cite

Objectives To assess the appropriateness of the statistical methodology used in a recent meta-analysis investigating the effect of maternity waiting homes (MWHs) on perinatal mortality in Sub-Saharan Africa. Results A recent meta-analysis published in BMC Research Notes used a fixed-effect model to generate an unadjusted summary estimate of the effectiveness of MWHs in reducing perinatal mortality in Africa using ten observational studies (pooled odds ratio 0.15, 95% confidence interval 0.14–0.17). The authors concluded that MWHs reduce perinatal mortality by over 80% and should be incorporated into routine maternal health care services. In the present article, we illustrate that due to the contextual and methodological heterogeneity present in existing studies, the authors’ conclusions about the effectiveness of MWHs in reducing perinatal mortality were likely overstated. Additionally, we argue that because of the selection bias and confounding inherent in observational studies, unadjusted pooled estimates provide little causal evidence for effectiveness. Additional studies with robust designs are required before an appropriately designed meta-analysis can be conducted; until then, the ability to draw causal inferences regarding the effectiveness of MWHs in reducing perinatal mortality is limited.

show abstract

“…Second, considering the substantial heterogeneity in observational studies, it is not always appropriate to combine the results through meta-analysis (2,3). Both clinical and methodologic heterogeneity should be thoroughly discussed.…”

mentioning

confidence: 99%

Challenges in Systematic Reviews That Include Observational Studies: Comment on the Article by Karmacharya et al

Luo

Funada

Yoshioka

et al. 2020

Arthritis & Rheumatology

View full text Add to dashboard Cite

| 2163 in JNK-knockout mice was associated with increased expression of the cell senescence marker p16 INK4a in the synovium and cartilage. Despite the negative findings we reported from our study, Dr. Xu and Ms Huang questioned whether JNK signaling might still be a valuable target to treat OA. They noted conflicting results from previously published in vitro studies that demonstrated that JNK inhibition reduced catabolic and proinflammatory signaling, as well as findings from a prior study indicating that JNK2 deletion reduced cartilage damage in the DMM model. At issue is the very complex nature of signal transduction pathways, such as the JNK pathway, that serve multiple homeostatic and potentially pathologic functions. In a previous study (1), our group noted that Col2Cre-mediated conditionaldouble-knockout of JNK1/2 in type II collagen-expressing cells resulted in a severe scoliotic phenotype, indicating that JNK signaling is necessary for normal spinal development. This finding was supported by a report of a similar scoliotic phenotype in mice in which c-Jun, which is activated by JNK, was knocked out (2). In our present study, the finding of increased cell senescence in JNK-knockout mice was surprising; however, it was supported by findings of previous studies in non-joint tissues (3,4). A second issue is that in vitro experiments using a single cell type in culture, such as those with results suggesting that JNK inhibition could inhibit catabolic pathways involved in OA, do not necessarily translate to the complexity of the in vivo environment. A particular issue in many studies of JNK signaling has been the use of JNK inhibitors, such as SP600125, that have been found to lack specificity and have off-target effects at concentrations used to inhibit JNK (5). A third issue is the lack of replication of findings of in vivo studies in mice, even when the same mouse strain from the same source is used. One important consideration is the number of mice in each experimental group. In human studies, positive results in early-phase studies using a small number of participants are often not replicated in larger clinical trials. In the report by Ismail et al (6) in which the authors noted less cartilage damage after DMM surgery in JNK-knockout mice, only 4-6 mice per experimental group were studied, whereas we studied 15 mice per experimental group. We agree that further study of the role of JNK signaling in articular joints is needed. In particular, we are interested in determining the mechanism by which JNK loss leads to cell senescence. However, we would argue that the preclinical evidence to date does not support JNK inhibition as a safe and effective intervention for OA in humans.

show abstract

Challenges in meta-analyses with observational studies

Cited by 191 publications

References 22 publications

Can statistical adjustment guided by causal inference improve the accuracy of effect estimation? A simulation and empirical research based on meta-analyses of case–control studies

Can statistical adjustment guided by causal inference improve the accuracy of effect estimation? A simulation and empirical research based on meta-analyses of case–control studies

The effect of maternity waiting homes on perinatal mortality is inconclusive: a critical appraisal of existing evidence from Sub-Saharan Africa

Challenges in Systematic Reviews That Include Observational Studies: Comment on the Article by Karmacharya et al

Contact Info

Product

Resources

About