Floodgate: inference for model-free variable importance

Zhang, Lu; Janson, Lucas

doi:10.48550/arxiv.2007.01283

Cited by 5 publications

(8 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Kpc: the Population Versionmentioning

confidence: 99%

“…Remark 2.5 (Connection to Zhang and Janson [150]). When d = 1, the numerator of ρ 2 (Y, Z|X) is equal to the minimum mean squared error gap (mMSE gap): 2 , which has been used to quantify the conditional relationship between Y and Z given X in the recent paper [150]. Note that mMSE gap is not invariant under arbitrary scalings of Y , but ρ 2 (Y, Z|X) (which is equal to the squared partial correlation ρ 2 Y Z•X ; see Proposition 2.1) is.…”

Section: Kpc: the Population Versionmentioning

confidence: 99%

See 1 more Smart Citation

Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence

Huang¹,

Deb²,

Sen³

2020

Preprint

View full text Add to dashboard Cite

In this paper we propose and study a class of simple, nonparametric, yet interpretable measures of conditional dependence between two random variables Y and Z given a third variable X, all taking values in general topological spaces. The population version of any of these nonparametric measures -defined using the theory of reproducing kernel Hilbert spaces (RKHSs) -captures the strength of conditional dependence and it is 0 if and only if Y and Z are conditionally independent given X, and 1 if and only if Y is a measurable function of Z and X. Thus, our measure -which we call kernel partial correlation (KPC) coefficient -can be thought of as a nonparametric generalization of the classical partial correlation coefficient that possesses the above properties when (X, Y, Z) is jointly normal. We describe two consistent methods of estimating KPC. Our first method of estimation is graph-based and utilizes the general framework of geometric graphs, including K-nearest neighbor graphs and minimum spanning trees. A sub-class of these estimators can be computed in near linear time and converges at a rate that automatically adapts to the intrinsic dimensionality of the underlying distribution(s). Our second strategy involves direct estimation of conditional mean embeddings using cross-covariance operators in the RKHS framework. Using these empirical measures we develop forward stepwise (high-dimensional) nonlinear variable selection algorithms. We show that our algorithm, using the graph-based estimator, yields a provably consistent model-free variable selection procedure, even in the high-dimensional regime when the number of covariates grows exponentially with the sample size, under suitable sparsity assumptions. Extensive simulation and real-data examples illustrate the superior performance of our methods compared to existing procedures. The recent conditional dependence measure proposed by Azadkia and Chatterjee [5] can also be viewed as a special case of our general framework.

show abstract

Section: Kpc: the Population Versionmentioning

confidence: 99%

Section: Kpc: the Population Versionmentioning

confidence: 99%

Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence

Huang¹,

Deb²,

Sen³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Examples include pre-validation (Tibshirani and Efron, 2002;Höfling and Tibshirani, 2008) and cross-fitting (e.g, Newey and Robins, 2018)). Further, confidence intervals for prediction accuracy are used to evaluate variable importance (Williamson et al, 2021;Zhang and Janson, 2020). We suspect that our nested cross-validation proposal could be adapted to improve the accuracy of these and related approaches.…”

Section: Discussionmentioning

confidence: 99%

Cross-validation: what does it estimate and how well does it do it?

Bates¹,

Hastie²,

Tibshirani³

2021

Preprint

View full text Add to dashboard Cite

Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's Cp. Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not re-fit the model on the combined data, since this invalidates the confidence intervals.

show abstract

“…Some propose model class reliance (MCR) [21,60] that investigate the feature importance across a set of well-performed models on the same data set, instead of one particular model in use. More related approaches that construct confidence intervals for feature importance include CPI [75], GCM [57], Floodgate [86], and VIME [78,77]. Many are only applicable for regression tasks and not classification, others make specific distributional assumptions that might not hold for general ML problems.…”

Section: Related Workmentioning

confidence: 99%

Model-Agnostic Confidence Intervals for Feature Importance: A Fast and Powerful Approach Using Minipatch Ensembles

Gan¹,

Zheng²,

Allen³

2022

Preprint

View full text Add to dashboard Cite

In order to trust machine learning for high-stakes problems, we need models to be both reliable and interpretable. Recently, there has been a growing body of work on interpretable machine learning which generates human understandable insights into data, models, or predictions. At the same time, there has been increased interest in quantifying the reliability and uncertainty of machine learning predictions, often in the form of confidence intervals for predictions using conformal inference. Yet, there has been relatively little attention given to the reliability and uncertainty of machine learning interpretations, which is the focus of this paper. Our goal is to develop confidence intervals for a widely-used form of machine learning interpretation: feature importance. We specifically seek to develop universal model-agnostic and assumption-light confidence intervals for feature importance that will be valid for any machine learning model and for any regression or classification task. We do so by leveraging a form of random observation and feature subsampling called minipatch ensembles and show that our approach provides assumption-light asymptotic coverage for the feature importance score of any model. Further, our approach is fast as computations needed for inference come nearly for free as part of the ensemble learning process. Finally, we also show that our same procedure can be leveraged to provide valid confidence intervals for predictions, hence providing fast, simultaneous quantification of the uncertainty of both model predictions and interpretations. We validate our intervals on a series of synthetic and real data examples, showing that our approach detects the correct important features and exhibits many computational and statistical advantages over existing methods. * Equal contribution.

show abstract

Floodgate: inference for model-free variable importance

Cited by 5 publications

References 73 publications

Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence

Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence

Cross-validation: what does it estimate and how well does it do it?

Model-Agnostic Confidence Intervals for Feature Importance: A Fast and Powerful Approach Using Minipatch Ensembles

Contact Info

Product

Resources

About