Can We Trust Observational Studies Using Propensity Scores in the Critical Care Literature? A Systematic Comparison With Randomized Clinical Trials*

Abstract

Across diverse critical care topics, propensity score studies published in high-impact journals produced results that were generally consistent with the findings of randomized clinical trials. However, caution is needed when interpreting propensity score studies because occasionally their results contradict those of randomized clinical trials and there is no reliable way to predict disagreements.

Comparison of treatment effect estimates of non-vitamin K antagonist oral anticoagulants versus warfarin between observational studies using propensity score methods and randomized controlled trials

“…In our study, we found general consistency of the treatment effect estimates for NOACs between PS studies and RCTs. Notwithstanding, while some studies matched PS studies with RCTs in terms of populations, interventions, and outcomes [24,[26][27][28][29], the matching approach was not used in our study. First, our relatively narrow research question was clearly defined with similar populations (patients with AF), interventions (NOACs), and outcomes (stroke or systemic embolism, and major bleeding) in the PS studies and RCTs.…”

“…Nevertheless, no significant difference was found between the PS-and RCT-derived estimates in the subgroup analyses by doses of NOACs, or by different components of the composite outcomes (Table 3). Several systematic reviews have compared the treatment effect estimates from PS studies and RCTs in different clinical settings, with a conclusion that PS studies may either overestimate [24,26], underestimate [27], or be consistent with [25,28,29] the estimates from RCTs. Therefore the concordance between PS studies and RCTs may depend on different populations, interventions and outcomes.…”

“…Recent cohort studies have used the PS method to obtain treatment effect estimates from observational data. Several systematic reviews compare the results from PS studies and RCTs in different populations, yielding different conclusions [24][25][26][27][28][29]. However, no comparison of treatment effect estimates of NOACs versus warfarin in patients with AF from PS studies and RCTs is available.…”

A propensity score matched analysis of obesity as an independent risk factor for postoperative complications in reduction mammaplasty

“…While multivariable analysis attempts to control for comorbidities via advanced statistical techniques, 1:1 matching is a dramatically more powerful technique that matches each study patient with a near-identical "control" patient, in spite of detractors of this technique. [24] After analysis of matched cohorts, only wound complications were increased in the obese population [ Table 8]. On further analysis, the difference was mainly attributed to a risk of increased surgical site infection in the obese cohort.…”

Challenges and Opportunities in Secondary Analyses of Electronic Health Record Data

“…Kitsios and colleagues matched 18 unique propensity score studies in the ICU setting with at least one RCT evaluating the same clinical question and found a high degree of agreement between their estimates of relative risk and effect size. There was substantial difference in the magnitude of effect sizes in a third of comparisons, reaching statistically significance in one case [27]. Though the RCT remains atop the hierarchy of evidence-based medicine, it is hard to ignore the power of large observational studies that include adequate adjusting for covariates, such as carefully performed studies derived from review of EHRs.…”