Panning for Gold: ‘Model-X’ Knockoffs for High Dimensional Controlled Variable Selection

Abstract

Summary Many contemporary large‐scale applications involve building interpretable models linking a large set of potential covariates to a response in a non‐linear fashion, such as when the response is binary. Although this modelling problem has been extensively studied, it remains unclear how to control the fraction of false discoveries effectively even in high dimensional logistic regression, not to mention general high dimensional non‐linear models. To address such a practical problem, we propose a new framework of ‘model‐X’ knockoffs, which reads from a different perspective the knockoff procedure that was originally designed for controlling the false discovery rate in linear models. Whereas the knockoffs procedure is constrained to homoscedastic linear models with n⩾p, the key innovation here is that model‐X knockoffs provide valid inference from finite samples in settings in which the conditional distribution of the response is arbitrary and completely unknown. Furthermore, this holds no matter the number of covariates. Correct inference in such a broad setting is achieved by constructing knockoff variables probabilistically instead of geometrically. To do this, our approach requires that the covariates are random (independent and identically distributed rows) with a distribution that is known, although we provide preliminary experimental evidence that our procedure is robust to unknown or estimated distributions. To our knowledge, no other procedure solves the controlled variable selection problem in such generality but, in the restricted settings where competitors exist, we demonstrate the superior power of knockoffs through simulations. Finally, we apply our procedure to data from a case–control study of Crohn's disease in the UK, making twice as many discoveries as the original analysis of the same data.

On the construction of knockoffs in case–control studies

“…In other words, implies that a variable selection procedure that is likely to mistakenly select irrelevant variable X j is equally likely to select the constructed knockoff feature , which then alerts us to the fact that our variable selection procedure is selecting false positives. We refer the reader to the already extensive literature on the subject, for example, Barber and Candès () and Candès et al (), for further information.…”

Deep latent variable models for generating knockoffs

“…A recent breakthrough in the statistical theory, that is, the model‐X framework (Candès et al, ), provides a general solution for the FDR controlled variable selection. It can be incorporated with any machine learning models to select the true signals associated with the outcome, with rigorous control for FDR.…”

“…It can be incorporated with any machine learning models to select the true signals associated with the outcome, with rigorous control for FDR. However, the implementation of model‐X requires the generation of the so‐called “knockoffs,” where there are limited existing methods (Candès et al, ). The goal of our paper is to fill in the gap to make the model‐X framework applicable for features X from a flexible distribution.…”

“…The requirement for the feature statistic is that for the null variables ( H 0 j is the true null): (a) the distribution for the W j s is symmetrically distributed with respect to 0; and (b) conditioning on the | W j |, the signs of W j s are independent. Based on exchangeability, Candès et al () provide a way to construct W j through for some feature summary statistics V j and its counterpart summary statistics for the knockoff feature . An example of such feature statistics is the signed‐max‐lambda statistics in L 1 penalized regressions (Barber & Candès, ): for the concatenated design matrix , define where is a predetermined loss function.…”

Global error control procedure for spatially structured targets

“…COMET elaborates on both a recent alternative class of FDR controlling procedures [5], [6] and a recent method [7] that yields robust control under weak assumptions on the noise. We propose a relevant framework in the context of multiple-testing problem where target exhibit connected structures.…”