Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison

Pohl, Steffi; Steiner, Péter; Eisermann, Jens; Soellner, Renate; Cook, Thomas D.

doi:10.3102/0162373709343964

Cited by 52 publications

(51 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PS ϭ propensity score; ANCOVA ϭ analysis of covariance. Pohl, Steiner, Eisermann, Soellner, and Cook (2009) also succeeded in removing all the selection bias in the quasi-experiment despite a different selection mechanism and different degrees of initial bias. Additional analyses of this data set also indicated that proxy-pretest measures and topic preference were once again necessary and sufficient for removing all the selection bias .…”

Section: Discussionmentioning

confidence: 98%

The importance of covariate selection in controlling for selection bias in observational studies.

Steiner¹,

Cook²,

Shadish³

et al. 2010

Psychological Methods

Self Cite

404

306

View full text Add to dashboard Cite

The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known.

show abstract

Section: Discussionmentioning

confidence: 98%

The importance of covariate selection in controlling for selection bias in observational studies.

Steiner¹,

Cook²,

Shadish³

et al. 2010

Psychological Methods

Self Cite

404

306

View full text Add to dashboard Cite

show abstract

“…Here, we make use of an ANCOVA method (Mayer, Dietzfelbinger, & Rosseel, ; Steyer & Partchev, ) and a PS method (Lockwood & McCaffrey, ) that can incorporate latent covariates for adjustment. We apply these methods, using empirical data from a within‐study comparison (WSC; Pohl, Steiner, Eisermann, Soellner, & Cook, ) and evaluate whether attenuation bias occurs in empirical data. In the following, we first review the theoretical results on attenuation bias and describe the adjustment methods.…”

Section: Impact Of Adjusting For Fallible or Latent Covariates In Pramentioning

confidence: 99%

“…We use the data of a four‐arm WSC conducted by Pohl et al . () and evaluate the accuracy of ATE estimates when we correct for measurement error in covariates in comparison to using fallible covariates for adjustment. In the quasi‐experimental data, the participants self‐selected to the treatment conditions and had no information about the fallible covariates.…”

Section: Impact Of Adjusting For Fallible or Latent Covariates In Pramentioning

confidence: 99%

When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration

Sengewald

Steiner

Pohl

2018

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

The average causal treatment effect (ATE) can be estimated from observational data based on covariate adjustment. Even if all confounding covariates are observed, they might not necessarily be reliably measured and may fail to obtain an unbiased ATE estimate. Instead of fallible covariates, the respective latent covariates can be used for covariate adjustment. But is it always necessary to use latent covariates? How well do analysis of covariance (ANCOVA) or propensity score (PS) methods estimate the ATE when latent covariates are used? We first analytically delineate the conditions under which latent instead of fallible covariates are necessary to obtain the ATE. Then we empirically examine the difference between ATE estimates when adjusting for fallible or latent covariates in an applied example. We discuss the issue of fallible covariates within a stochastic theory of causal effects and analyse data of a within-study comparison with recently developed ANCOVA and PS procedures that allow for latent covariates. We show that fallible covariates do not necessarily bias ATE estimates, but point out different scenarios in which adjusting for latent covariates is required. In our empirical application, we demonstrate how latent covariates can be incorporated for ATE estimation in ANCOVA and in PS analysis.Impact of measurement error in covariates 245

show abstract

“…As discussed above, Shadish, Clark, and Steiner (2008) like setting, and short intervention period. Pohl, Steiner, Eisermann, Soellner, and Cook (2009) conducted a replication of the WSC approach used by Shadish et al (2008) Demographic Covariates. In many observational studies, the carefully considered selection process and rich covariate sets exemplified by Shadish et al (2008) and Pohl et al (2009) are the exceptions rather than the rule.…”

Section: Performance Of Local Versus Non-local Comparison Groupsmentioning

confidence: 99%

“…Pohl, Steiner, Eisermann, Soellner, and Cook (2009) conducted a replication of the WSC approach used by Shadish et al (2008) Demographic Covariates. In many observational studies, the carefully considered selection process and rich covariate sets exemplified by Shadish et al (2008) and Pohl et al (2009) are the exceptions rather than the rule. Observational studies often depend on simple predictors of convenience such as race/ethnicity, gender, free-reduced price lunch status, disability status, and gender.…”

Section: Performance Of Local Versus Non-local Comparison Groupsmentioning

confidence: 99%

Empirical Performance of Covariates in Education Observational Studies

Wong

Valentine

Miller-Bains

2016

Journal of Research on Educational Effectiveness

View full text Add to dashboard Cite

This paper summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether pre-intervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The paper Downloaded by [University of Nebraska, Lincoln] at 05:40 13 June 2016 ACCEPTED MANUSCRIPT ACCEPTED MANUSCRIPT 2 concludes by offering education researchers empirically-based guidance on covariate selection in observational studies.

show abstract

Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison

Cited by 52 publications

References 51 publications

The importance of covariate selection in controlling for selection bias in observational studies.

The importance of covariate selection in controlling for selection bias in observational studies.

When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration

Empirical Performance of Covariates in Education Observational Studies

Contact Info

Product

Resources

About