A Conditional Randomization Test to Account for Covariate Imbalance in Randomized Experiments

Hennessy, Jonathan; Dasgupta, Tirthankar; Miratrix, Luke; Pattanayak, Cassandra Wolos; Sarkar, Pradipta

doi:10.1515/jci-2015-0018

Cited by 32 publications

(49 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This gives a conditional randomization test. Zheng and Zelen (2008) and Hennessy et al (2016) demonstrated that conditional randomization tests often improve the power as long as the covariates are predictive of the outcomes. Holt and Smith (1979) and Miratrix et al (2013) discussed post-stratification, the estimation analog of testing.…”

Section: The Above Null Hypothesis Must Satisfymentioning

confidence: 99%

Randomization Tests for Weak Null Hypotheses in Randomized Experiments

Ding

2020

Journal of the American Statistical Association

View full text Add to dashboard Cite

The Fisher randomization test (FRT) is appropriate for any test statistic, under a sharp null hypothesis that can recover all missing potential outcomes. However, it is often of interest to test a weak null hypothesis that the treatment does not affect the units on average. To use the FRT for a weak null hypothesis, we must address two issues. First, we need to impute the missing potential outcomes although the weak null hypothesis cannot determine all of them. Second, we need to choose a suitable test statistic. For a general weak null hypothesis, we propose an approach to imputing missing potential outcomes under a compatible sharp null hypothesis. With this imputation scheme, we advocate using a studentized statistic. The resulting FRT has multiple desirable features. First, it is model-free. Second, it is finite-sample exact under the sharp null hypothesis that we use to impute the potential outcomes. Third, it preserves correct large-sample type I errors under the weak null hypothesis of interest. Therefore, our FRT is agnostic to treatment effect heterogeneity. We establish a unified theory for general factorial experiments. We also extend it to stratified and clustered experiments.

show abstract

Section: The Above Null Hypothesis Must Satisfymentioning

confidence: 99%

Randomization Tests for Weak Null Hypotheses in Randomized Experiments

Ding

2020

Journal of the American Statistical Association

View full text Add to dashboard Cite

show abstract

“…(2) the true propensity score model is a logistic regression model, and (3) the collection of covariates is sufficient for the logistic regression model. More recently, Hennessy et al (2016) proposed a conditional randomization test for randomized experiments that is similar to Rosenbaum (1984) in that it also permutes within groups of units with the same covariate values, but it does not require any kind of model specification. Rosenbaum (1984) and Hennessy et al (2016) only consider cases with categorical covariates, and they make connections between their randomization tests and adjustment methods for categorical covariates, such as post-stratification (Miratrix et al, 2013).…”

Section: Accounting For Covariate Balance In Randomization Testsmentioning

confidence: 99%

Randomization Tests that Condition on Non-Categorical Covariate Balance

Branson

Miratrix

2019

Journal of Causal Inference

View full text Add to dashboard Cite

A benefit of randomized experiments is that covariate distributions of treatment and control groups are balanced on average, resulting in simple unbiased estimators for treatment effects.However, it is possible that a particular randomization yields covariate imbalances that researchers want to address in the analysis stage through adjustment or other methods. Here we present a randomization test that conditions on covariate balance by only considering treatment assignments that are similar to the observed one in terms of covariate balance.Previous conditional randomization tests have only allowed for categorical covariates, while our randomization test allows for any type of covariate. Through extensive simulation studies, we find that our conditional randomization test is more powerful than unconditional randomization tests and other conditional tests. Furthermore, we find that our conditional randomization test is valid (1) unconditionally across levels of covariate balance, and (2) conditional on particular levels of covariate balance. Meanwhile, unconditional randomization tests are valid for (1) but not (2). Finally, we find that our conditional randomization test is similar to a randomization test that uses a model-adjusted test statistic. * We would like to thank two anonymous reviewers and the Associate Editor, Peter Aronow, for their insightful comments that led to notable improvements in this work.

show abstract

“…() for network dependence and Hennessy et al . () for covariate imbalance. In such cases it is sometimes possible to develop non‐uniform randomization schemes that result in valid p ‐values, as with the CPT and the CRT.…”

Section: Introductionmentioning

confidence: 99%

The Conditional Permutation Test for Independence While Controlling for Confounders

Berrett

Wang

Barber

et al. 2019

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

Summary We propose a general new method, the conditional permutation test, for testing the conditional independence of variables X and Y given a potentially high dimensional random vector Z that may contain confounding factors. The test permutes entries of X non‐uniformly, to respect the existing dependence between X and Z and thus to account for the presence of these confounders. Like the conditional randomization test of Candès and co‐workers in 2018, our test relies on the availability of an approximation to the distribution of X|Z—whereas their test uses this estimate to draw new X‐values, for our test we use this approximation to design an appropriate non‐uniform distribution on permutations of the X‐values already seen in the true data. We provide an efficient Markov chain Monte Carlo sampler for the implementation of our method and establish bounds on the type I error in terms of the error in the approximation of the conditional distribution of X|Z, finding that, for the worst‐case test statistic, the inflation in type I error of the conditional permutation test is no larger than that of the conditional randomization test. We validate these theoretical results with experiments on simulated data and on the Capital Bikeshare data set.

show abstract

A Conditional Randomization Test to Account for Covariate Imbalance in Randomized Experiments

Cited by 32 publications

References 35 publications

Randomization Tests for Weak Null Hypotheses in Randomized Experiments

Randomization Tests for Weak Null Hypotheses in Randomized Experiments

Randomization Tests that Condition on Non-Categorical Covariate Balance

The Conditional Permutation Test for Independence While Controlling for Confounders

Contact Info

Product

Resources

About