A modern maximum-likelihood theory for high-dimensional logistic regression

Abstract

Students in statistics or data science usually learn early on that when the sample size n is large relative to the number of variables p, fitting a logistic model by the method of maximum likelihood produces estimates that are consistent and that there are well-known formulas that quantify the variability of these estimates which are used for the purpose of statistical inference. We are often told that these calculations are approximately valid if we have 5 to 10 observations per unknown parameter. This paper shows that this is far from the case, and consequently, inferences produced by common software packages are often unreliable. Consider a logistic model with independent features in which n and p become increasingly large in a fixed ratio. We prove that (i) the maximum-likelihood estimate (MLE) is biased, (ii) the variability of the MLE is far greater than classically estimated, and (iii) the likelihood-ratio test (LRT) is not distributed as a χ2. The bias of the MLE yields wrong predictions for the probability of a case based on observed values of the covariates. We present a theory, which provides explicit expressions for the asymptotic bias and variance of the MLE and the asymptotic distribution of the LRT. We empirically demonstrate that these results are accurate in finite samples. Our results depend only on a single measure of signal strength, which leads to concrete proposals for obtaining accurate inference in finite samples through the estimate of this measure.

Renewable Estimation and Incremental Inference in Generalized Linear Models with Streaming Data Sets

“…We set N B =2×10 5 , B =20 and , and simulate p ‐element vectors of covariates from . Following Sur and Candés (), to guarantee the existence of the MLE in such high dimensional settings, we generate the true values of entrywise IID from under p =1000 and from under p =2500. The same criteria are used in the subsequent assessment and comparisons.…”

Hierarchical inference for genome-wide association studies: a view on methodology with software

“…In the hierarchical case though, this remains unclear. In addition, the p-values from multi-sample splitting, as used in our procedure and software, might be unreliable: it is challenging, in particular for logistic regression (Sur and Candès, 2018), to come up with reliable and powerful p-values for testing single or groups of regression coefficients which are reliable and powerful in high-dimensional settings. (3) The role of the hierarchy is an issue of power, as long as we assume fixed design and a correct model specification.…”

Sharp Guarantees for Solving Random Equations with One-Bit Information

“…Finally, our paper is closely related to [7], [28], in which the authors study the high-dimensional performance of maximum-likelihood (ML) estimation for the logistic model. The ML estimator is a special case of (1) but their measurement model differs from the one considered in this paper.…”