Metrics for covariate balance in cohort studies of causal effects

Abstract

Inferring causation from non-randomized studies of exposure requires that exposure groups can be balanced with respect to prognostic factors for the outcome. Although there is broad agreement in the literature that balance should be checked, there is confusion regarding the appropriate metric. We present a simulation study that compares several balance metrics with respect to the strength of their association with bias in estimation of the effect of a binary exposure on a binary, count, or continuous outcome. The simulations utilize matching on the propensity score with successively decreasing calipers to produce datasets with varying covariate balance. We propose the post-matching C-statistic as a balance metric and found that it had consistently strong associations with estimation bias, even when the propensity score model was misspecified, as long as the propensity score was estimated with sufficient study size. This metric, along with the average standardized difference and the general weighted difference, outperformed all other metrics considered in association with bias, including the unstandardized absolute difference, Kolmogorov-Smirnov and Lévy distances, overlapping coefficient, Mahalanobis balance, and L1 metrics. Of the best-performing metrics, the C-statistic and general weighted difference also have the advantage that they automatically evaluate balance on all covariates simultaneously and can easily incorporate balance on interactions among covariates. Therefore, when combined with the usual practice of comparing individual covariate means and standard deviations across exposure groups, these metrics may provide useful summaries of the observed covariate imbalance.

Sulfonylureas and risk of falls and fractures among nursing home residents with type 2 diabetes mellitus

“…We matched as many as 10 biguanide monotherapy users to each combination therapy user. We used the c-statistic to assess the degree of overlap between exposure groups (c-statistic monotherapy = 0.67; c-statistic combination therapy = 0.70) [32] and noted that none of the potential confounders differed by greater than 3.6% absolute difference between the comparator groups. Cox models accounting for the propensity-matched design were used to estimate adjusted hazards ratios (aHR) and 95% confidence intervals (CI).…”

Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: a systematic review

“…Recently, postmatching C-statistic of the PS model [51] has been suggested as an overall measure of the balance across covariates. It may be simpler to evaluate and has shown comparable performance with SDif in terms of indicating bias; however, unlike the SDif, an assessment of covariate's potential for confounding (by checking balance on the covariate's scale) and identification of whether the imbalances are due to a set of related covariates are difficult [21,38,51]. An iterative process of fitting the PS model, checking balance on covariates, and respecifying the PS model has been suggested by Rosenbaum and Rubin [2].…”

Endoscopic resection of high-risk T1 colorectal carcinoma prior to surgical resection has no adverse effect on long-term outcomes

“…First, we evaluated the improvement in baseline comparability between the two surgical approaches before and after IPW adjustment for confounders. Next, we assessed the c-index of the propensity models from step 1, after weighting patients by their propensity score-derived IPW (a c-index of 0.5 means that the resulting IPW model cannot discriminate which patient received which treatment, indicating successfully obtained balance; a c-index of 1.0 on the other hand indicates extreme remaining imbalance) 23. As extreme weights for some individuals can lead to bias and loss of precision in treatment effect estimates,24 we inspected the distribution of weights and conducted sensitivity analyses truncating weights at the 97.5th percentile.…”