Insights on Variance Estimation for Blocked and Matched Pairs Designs

Pashley, Nicole E.; Miratrix, Luke

doi:10.3102/1076998620946272

Cited by 28 publications

(30 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This allows us to show which overall frameworks guarantee no harm from blocking and which do not. We focus on comparing true variance rather than delving into the problem of variance estimation (see Pashley & Miratrix, 2021, for discussion of variance estimation). For the superpopulation contexts, we also carefully separate out variance due to the sampling of units from variance due to randomized treatment assignment; this gives more precise statements of the benefits of blocking than have been given in prior literature.…”

Section: Many Of Us May Have Embraced George Box's Famous Quote 1 Ever Since It Was Thrown At Us During An Undergraduate Statistics Coursmentioning

confidence: 99%

“…Under a completely randomized design, n t ¼ np of the units are randomly assigned to treatment, with the rest of the n c ¼ n À n t units assigned to control, for fixed p 2 ð0; 1Þ. Under a blocked design, the units are split into K blocks in some manner (see Pashley & Miratrix, 2021, for longer discussion of block types), with n k units in the kth block (k ¼ 1; : : : ; K). Within each block, a completely randomized experiment is performed independently of other blocks such that in the kth block, n t;k ¼ p k n k are randomly assigned to treatment for fixed p k 2 ð0; 1Þ.…”

Section: Experimental Designs and Estimandsmentioning

confidence: 99%

See 1 more Smart Citation

Block What You Can, Except When You Shouldn’t

Pashley

Miratrix

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this article, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of estimator precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions; for instance, we show that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking.

show abstract

Section: Many Of Us May Have Embraced George Box's Famous Quote 1 Ever Since It Was Thrown At Us During An Undergraduate Statistics Coursmentioning

confidence: 99%

Section: Experimental Designs and Estimandsmentioning

confidence: 99%

Block What You Can, Except When You Shouldn’t

Pashley

Miratrix

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several authors have compared the variance of the simple difference estimator for completely and pair randomized designs under the Neyman–Rubin model (e.g., see Imai, 2008; Pashley and Miratrix, 2017). The difference in variance under these designs can be either positive or negative.…”

Section: Imputation Methods Of Potential Differences In Paired Expmentioning

confidence: 99%

Design-Based Covariate Adjustments in Paired Experiments

Gagnon-Bartsch

2020

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

In paired experiments, participants are grouped into pairs with similar characteristics, and one observation from each pair is randomly assigned to treatment. The resulting treatment and control groups should be well-balanced; however, there may still be small chance imbalances. Building on work for completely randomized experiments, we propose a design-based method to adjust for covariate imbalances in paired experiments. We leave out each pair and impute its potential outcomes using any prediction algorithm such as lasso or random forests. This method addresses a unique trade-off that exists for paired experiments. By addressing this trade-off, the method has the potential to improve precision over existing methods.

show abstract

“…For example, Miratrix et al (2013) discusses Neymanian inference for blocked experiments, and Imai (2008) does the same for paired experiments. See Pashley & Miratrix (2017) for a discussion of variance estimation for these two designs as well as hybrid designs that involve blocks and pairs. Neymanian inference has also been established for factorial designs (Dasgupta et al, 2015), rerandomized experiments (Li et al, 2018b), and their combination .…”

Section: The Analysis Stage: After Assuming An Experimental Design For Matched Datamentioning

confidence: 99%

Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

Branson¹

2021

Observational Studies

View full text Add to dashboard Cite

Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance. It is widely agreed upon that covariate balance can serve as evidence that a matched dataset approximates a randomized experiment, but what kind of experiment does a matched dataset approximate? In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization. Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design-if any-is most appropriate for a matched dataset. After researchers determine a plausible design, we recommend a randomizationbased approach for analyzing the matched data, which can incorporate any design and treatment effect estimator. Through simulation, we find that our test can frequently detect violations of randomized assignment that harm inferential results. Furthermore, through simulation and a real application in political science, we find that matched datasets with high levels of covariate balance tend to approximate balance-constrained designs like rerandomization, and analyzing them as such can lead to precise causal analyses. However, assuming a precise design should be proceeded with caution, because it can harm inferential results if there are still substantial biases due to remaining imbalances after matching. Our approach is implemented in the randChecks R package, available on CRAN.

show abstract

Insights on Variance Estimation for Blocked and Matched Pairs Designs

Cited by 28 publications

References 31 publications

Block What You Can, Except When You Shouldn’t

Block What You Can, Except When You Shouldn’t

Design-Based Covariate Adjustments in Paired Experiments

Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

Contact Info

Product

Resources

About