Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance

Abstract

BackgroundDouble-adjustment can be used to remove confounding if imbalance exists after propensity score (PS) matching. However, it is not always possible to include all covariates in adjustment. We aimed to find the optimal imbalance threshold for entering covariates into regression.MethodsWe conducted a series of Monte Carlo simulations on virtual populations of 5,000 subjects. We performed PS 1:1 nearest-neighbor matching on each sample. We calculated standardized mean differences across groups to detect any remaining imbalance in the matched samples. We examined 25 thresholds (from 0.01 to 0.25, stepwise 0.01) for considering residual imbalance. The treatment effect was estimated using logistic regression that contained only those covariates considered to be unbalanced by these thresholds.ResultsWe showed that regression adjustment could dramatically remove residual confounding bias when it included all of the covariates with a standardized difference greater than 0.10. The additional benefit was negligible when we also adjusted for covariates with less imbalance. We found that the mean squared error of the estimates was minimized under the same conditions.ConclusionIf covariate balance is not achieved, we recommend reiterating PS modeling until standardized differences below 0.10 are achieved on most covariates. In case of remaining imbalance, a double adjustment might be worth considering.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-017-0338-0) contains supplementary material, which is available to authorized users.

Post‐endoscopic retrograde cholangiopancreatography pancreatitis in patients with asymptomatic common bile duct stones

“…C‐statistics were calculated to evaluate the balance of propensity scores between the two groups. The balance of the propensity score‐matched sample was checked by standardized mean difference, which should be < 0.10 and is not influenced by sample size . The IPTW and regression adjustment methods were also performed to ensure the robustness of results of the propensity score matching analysis .…”

Uric Acid in the Follow-Up Determines 30% Decline in Estimated GFR Over 2 Years: a Propensity Score Analysis

“…The Cox regression model was further adjusted for other outcome-associated baseline covariates including sex, age, diabetic nephropathy, eGFR, SBP, Alb, P, proteinuria and use of RASi [5]. The residual imbalance, if existed, could be adjusted for these confounders by this double adjustment [17]. Adjusted HRs showed virtually the similar values as crude HRs, indicating that residual imbalance was minimum in three thresholds (Table 3, Model 3).…”

“…The covariates used for adjustment were chosen based on our previous analysis and the standard Cox regression model in the present study; age, sex, diabetic nephropathy, eGFR, SBP, Alb, P, proteinuria and use of RASi [5]. The double adjustment in propensity score analysis is recommended to remove the residual imbalance especially when standardized difference > 0.1 [17].…”

“…Some of the covariates which may influence the renal event were doubly adjusted for to remove the residual imbalance [17].…”

“…The caliper size was first set at 0.25 times standard deviation of the propensity scores in the participants and gradually narrowed until reaching the standardized difference < 0.1 on most covariates, the value being proposed to meet the balance between treated and untreated groups [17][18][19][20]. The model assignment was estimated by c-statistics and the balance between two groups was checked by paired comparison tests and standardized differences of the 23 baseline covariates [21].…”