When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration

Abstract

The average causal treatment effect (ATE) can be estimated from observational data based on covariate adjustment. Even if all confounding covariates are observed, they might not necessarily be reliably measured and may fail to obtain an unbiased ATE estimate. Instead of fallible covariates, the respective latent covariates can be used for covariate adjustment. But is it always necessary to use latent covariates? How well do analysis of covariance (ANCOVA) or propensity score (PS) methods estimate the ATE when latent covariates are used? We first analytically delineate the conditions under which latent instead of fallible covariates are necessary to obtain the ATE. Then we empirically examine the difference between ATE estimates when adjusting for fallible or latent covariates in an applied example. We discuss the issue of fallible covariates within a stochastic theory of causal effects and analyse data of a within-study comparison with recently developed ANCOVA and PS procedures that allow for latent covariates. We show that fallible covariates do not necessarily bias ATE estimates, but point out different scenarios in which adjusting for latent covariates is required. In our empirical application, we demonstrate how latent covariates can be incorporated for ATE estimation in ANCOVA and in PS analysis.Impact of measurement error in covariates 245

Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy

“…We focused on the impacts of surrogacy violations in linear models. Recent research has considered how to adapt other common estimators of causal effects, such as weighting, matching, and doubly robust estimation, to the case of error-prone confounders (Kuroki & Pearl, 2014; Lockwood & McCaffrey, 2015, 2016; McCaffrey, Lockwood, & Setodji, 2013; Sengewald et al, 2019; Webb-Vargas, Rudolph, Lenis, Murakami, & Stuart, 2017; Yi et al, 2012). Most of these methods also rely on the assumption of surrogacy of the error-prone measures.…”

“…On the other hand, prior test scores may be among the variables used to assign students to current-year teachers. This would violate the surrogacy assumptions required for consistency of the EIV regression estimator and could cause estimators that ignore measurement error, such as OLS regression, to have smaller bias than methods that try to account for it (Sengewald, Steiner, & Pohl, 2019; Steiner & Kim, 2016). Lockwood and McCaffrey (2019) develop empirical tests about what information is used to assign students to teachers and find evidence that assignments are influenced by both prior test scores and unobserved variables related to prior achievement.…”

ANCOVA versus Change Score for the Analysis of Nonexperimental Two-Wave Data: A Structural Modeling Perspective

“…error can bias the estimates of the ANCOVA approach (Culpepper & Aguinis, 2011;Sengewald, Pohl, & Steiner, 2019). Moreover, one advantage of the change score approach is that it is not sensitive to measurement error in the pretest (e.g., Kim & Steiner, 2019).…”

“…In certain designs, individuals are assigned to the treatment according to their observed values on the pretest (Reichardt, 2019). In this case, the observed pretest values should be included in the ANCOVA approach, and correcting for measurement error inwould result in biased estimates of the treatment effect (Sengewald et al, 2019).…”

Causal Effects Based on Latent Variable Models

“…Since the regression coefficient of the fallible covariate is biased towards zero, it is also more difficult to identify covariates which significantly affect the treatment or the outcome when measurement error is not appropriately taken into account. Sengewald, Steiner, and Pohl (2018b) identified several conditions under which controlling for fallible covariates leads to substantial bias in average effects. Even in a randomized experiment, where omitting the latent covariate does not introduce bias, it may still be beneficial to consider the latent covariate, because it increases power.…”