New multivariate tests for assessing covariate balance in matched observational studies

Abstract

We propose new tests for assessing whether covariates in a treatment group and matched control group are balanced in observational studies. The tests exhibit high power under a wide range of multivariate alternatives, some of which existing tests have little power for. The asymptotic permutation null distributions of the proposed tests are studied and the p-values calculated through the asymptotic results work well in simulation studies, facilitating the application of the test to large data sets. The tests are illustrated in a study of the effect of smoking on blood lead levels. The proposed tests are implemented in an R package BalanceCheck.

Efficient Algorithms for Building Representative Matched Pairs with Enhanced Generalizability

“…Provided that no outcome data are viewed, researchers typically perform matching multiple times and select the design based on covariate balance. Recently, many formal diagnostic tests of covariate balance have been proposed (Chen & Small, 2022; Gagnon‐Bartsch & Shem‐Tov, 2019; Yu, 2021). Researchers could perform a formal diagnostic test to assess if there is any residual imbalance in observed covariates $$(\widetilde{\bm{x}}, \bm{x})$$ between two groups in the matched cohort; analogously, a formal test could be carried out to examine residual imbalance in common covariates $$\widetilde{\bm{x}}$$ between the matched group and the template.…”

Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

“…A rule of thumb widely appreciated and accepted in empirical comparative effectiveness research is that the absolute standardized mean differences (SMDs) of all covariates are less than 0.1, or one-tenth of one pooled standard deviation (Silber et al, 2001, Austin andStuart, 2015). Some authors have proposed formal statistical procedures that test the equality of multivariate covariate distributions in the treated and matched control groups; notable examples include Rosenbaum (2005), Hansen and Bowers (2008), Austin (2009), Small (2020), andYu (2020), among others. However, there is a gap between equality of covariate distributions in the treated and matched control groups and making the randomization assumption in the downstream, finite-sample randomization inference.…”

Limitation of permutation‐based differential correlation analysis

“…In addition to being easy to implement and avoid parametric modeling assumptions, permutation testing can ensure exact control of the type I error rate for all test statistics under the null hypothesis (Hoeffding, 1952;Lehmann et al, 2005). These attractive characteristics have led to this approach being used across a range of settings (Chen & Small, 2022;Song & Chen, 2022;Zhan et al, 2017). Despite these successes, a critical and often overlooked condition for permutation tests is the requirement for exchangeability under the null hypothesis.…”