Propensity Score Analysis with Unreliable Covariates: A comparison of Five Reliability-adjustment Methods

Abstract

Propensity score analysis (PSA) is often used by researchers to control for selection bias due to multiple covariates in quasi-experimental studies. However, covariates with low reliability have been shown to lead to biased treatment effects estimates in PSA. Latent variable analysis is a promising strategy to reduce the negative effects of observed variables’ measurement error. This Monte Carlo simulation study compared the performance of five methods to adjust propensity scores for unreliability. The results indicate that the latent variable model with inclusive factor score (PSIF) generated the lowest relative bias of treatment effect estimates, followed by propensity score estimation with structural equation model (PS-SEM). However, only PSIF provided unbiased treatment effect estimates across conditions with high, medium and low reliability. The results also show that evaluation of covariate balance can be misleading when there are unreliable covariates, because treatment effect estimates can be biased when covariate balanced is deemed adequate.