A review of the use of propensity score diagnostics in papers published in high-ranking medical journals

Abstract

Background: Propensity scores are widely used to deal with confounding bias in medical research. An incorrectly specified propensity score model may lead to residual confounding bias; therefore it is essential to use diagnostics to assess propensity scores in a propensity score analysis. The current use of propensity score diagnostics in the medical literature is unknown. The objectives of this study are to (1) assess the use of propensity score diagnostics in medical studies published in high-ranking journals, and (2) assess whether the use of propensity score diagnostics differs between studies (a) in different research areas and (b) using different propensity score methods. Methods: A PubMed search identified studies published in high-impact journals between Jan 1st 2014 and Dec 31st 2016 using propensity scores to answer an applied medical question. From each study we extracted information regarding how propensity scores were assessed and which propensity score method was used. Research area was defined using the journal categories from the Journal Citations Report. Results: A total of 894 papers were included in the review. Of these, 187 (20.9%) failed to report whether the propensity score had been assessed. Commonly reported diagnostics were p-values from hypothesis tests (36.6%) and the standardised mean difference (34.6%). Statistical tests provided marginally stronger evidence for a difference in diagnostic use between studies in different research areas (p = 0.033) than studies using different propensity score methods (p = 0.061). Conclusions: The use of diagnostics in the propensity score medical literature is far from optimal, with different diagnostics preferred in different areas of medicine. The propensity score literature may improve with focused efforts to change practice in areas where suboptimal practice is most common.

Propensity score matching in otolaryngologic literature: A systematic review and critical appraisal

“…We present the first systematic review of the otolaryngology literature to evaluate the use and reporting of PSM methodology. We found systematic underreporting of PSM methodological components in the otolaryngological literature, which is in accordance with findings of other studies in different fields [14,21,[41][42][43].…”

“…Indeed, one of the limitations of PSM is that it only controls for covariates included in the propensity score. Unlike true randomization, residual confounding can still bias results when important covariates are not identified a priori , or not available for use in the propensity score [ 43 , 45 ]. Additionally, following matching, it is often standard practice to assess any residual inter-group differences through reporting covariate balance.…”

Reappraisal of Non-vitamin K Antagonist Oral Anticoagulants in Atrial Fibrillation Patients: A Systematic Review and Meta-Analysis

“…Standardized differences are an attribute of the sample, independent of the sample size. It is easy to compute and understand and is the most commonly used diagnostic method to measure the balance of covariate distribution between treatment groups (36,38). In our current analysis, all of the 20 observational studies applied PSM or IPTW to balance the covariates between NOACs and warfarin regimen group.…”

Informing power and sample size calculations when using inverse probability of treatment weighting using the propensity score

“…These can be informed by previously published research, pilot studies, or by the investigators' clinical judgment. Granger and colleagues conducted a review of the use of propensity score diagnostics in papers published in the medical literature and noted which studies reported estimates of the c‐statistic of the propensity score model 25 . Similarly, Sturmer and colleagues reviewed the use of propensity score methods in the medical literature 26 .…”