Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach

Abstract

We propose a model-free framework for sensitivity analysis of individual treatment effects (ITEs), building upon ideas from conformal inference. For any unit, our procedure reports the Γ-value, a number which quantifies the minimum strength of confounding needed to explain away the evidence for ITE.Our approach rests on the reliable predictive inference of counterfactuals and ITEs in situations where the training data is confounded. Under the marginal sensitivity model of Tan ( 2006), we characterize the shift between the distribution of the observations and that of the counterfactuals. We first develop a general method for predictive inference of test samples from a shifted distribution; we then leverage this to construct covariate-dependent prediction sets for counterfactuals. No matter the value of the shift, these prediction sets (resp. approximately) achieve marginal coverage if the propensity score is known exactly (resp. estimated). We describe a distinct procedure also attaining coverage, however, conditional on the training data. In the latter case, we prove a sharpness result showing that for certain classes of prediction problems, the prediction intervals cannot possibly be tightened. We verify the validity and performance of the new methods via simulation studies and apply them to analyze real datasets.

Doubly Robust Calibration of Prediction Sets under Covariate Shift

“…The recent paper Jin et al (2021) proposed a robust conformal prediction algorithm that builds upon the weighted conformal inference method from Tibshirani et al (2019). The paper derives prediction sets that achieve marginal coverage if the propensity score is known exactly, allowing for latent covariate shift of a magnitude bounded by a known sensitivity parameter.…”

“…We further discuss this in the covariate shift setting of primary interest in Section 8. Yin et al (2021) also studied the sensitivity analysis of ITE using a similar approach as the first method proposed in Jin et al (2021) while their method of analysis offers a different perspective.…”

Selection by Prediction with Conformal p-values

“…Our methods build upon the conformal inference framework [62,63]. Although conformal-inferencebased methods have been developed for reliable uncertainty quantification in various problems [38,12,33,60,5,24], the theoretical guarantee usually concerns a single test point. However in many applications one might be interested in a batch of individuals and desire uncertainty quantification for multiple test samples simultaneously; in such situations, these methods are insufficient due to the complex dependence structure of test scores and p-values as well as multiplicity issues.…”

“…Example 2.9 (Counterfactual inference). Predicting counterfactuals [38,33] from randomized experiments in causal inference is another application whose setup is similar to Example 2.8. Under the potential outcomes framework [31], we let {X i , T i , Y i (1), Y i (0)} N i=1 be i.i.d.…”

“…More generally, inferring whether certain patients have benefited from the treatment also provides evidence on treatment effect heterogeneity. This is a counterfactual inference problem [38,33] in which one could predict the counterfactuals, i.e., what would happen should one takes an alternative option, by learning from the outcomes of patients under that option, and then compare the prediction to the realized outcomes. In this case, the set of those declared as having benefited from the treatment is informative if the FDR is controlled.…”