Testing Weak Nulls in Matched Observational Studies

Abstract

We develop sensitivity analyses for the sample average treatment effect in matched observational studies while allowing unit‐level treatment effects to vary. The methods may be applied to studies using any optimal without‐replacement matching algorithm. In contrast to randomized experiments and to paired observational studies, we show for general matched designs that over a large class of test statistics, any procedure bounding the worst‐case expectation while allowing for arbitrary effect heterogeneity must be unnecessarily conservative if treatment effects are actually constant across individuals. We present a sensitivity analysis which bounds the worst‐case expectation while allowing for effect heterogeneity, and illustrate why it is generally conservative if effects are constant. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and that is valid for testing the sample average effect under additional restrictions which may be deemed benign by practitioners. Simulations demonstrate that this alternative procedure results in a valid sensitivity analysis for the weak null hypothesis under a host of reasonable data‐generating processes. The procedures allow practitioners to assess robustness of estimated sample average treatment effects to hidden bias while allowing for effect heterogeneity in matched observational studies.

The Central Role of the Propensity Score in Sensitivity Analysis for Matched Observational Studies

“…Third, for testing Neyman's weak null (which allows arbitrarily heterogeneous treatment effects), we give the first valid sensitivity analysis procedure for matching with treatment doses. Our method is built on the proposed generalized Rosenbaum sensitivity bounds and large-scale mixed integer programming, which can be regarded as the generalization of the optimization-based sensitivity analysis for testing Neyman's weak null proposed in Fogarty (2016).…”

“…We here leverage mixed integer programming techniques to develop a sensitivity analysis with valid type-I error rate control under H weak λ 0 , which can be viewed as the generalization (to the non-binary treatment dose case) of an existing sensitivity analysis method for testing weak null hypothesis in matched observational studies with binary treatments, proposed in Fogarty (2016).…”