The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square

Abstract

Logistic regression is used thousands of times a day to fit data, predict future outcomes, and assess the statistical significance of explanatory variables. When used for the purpose of statistical inference, logistic models produce p-values for the regression coefficients by using an approximation to the distribution of the likelihood-ratio test. Indeed, Wilks' theorem asserts that whenever we have a fixed number p of variables, twice the log-likelihood ratio (LLR) 2Λ is distributed as a χ 2 k variable in the limit of large sample sizes n; here, χ 2 k is a chi-square with k degrees of freedom and k the number of variables being tested. In this paper, we prove that when p is not negligible compared to n, Wilks' theorem does not hold and that the chi-square approximation is grossly incorrect; in fact, this approximation produces p-values that are far too small (under the null hypothesis).Assume that n and p grow large in such a way that p/n → κ for some constant κ < 1/2. We prove that for a class of logistic models, the LLR converges to a rescaled chi-square, namely, 2Λ d → α(κ)χ 2 k , where the scaling factor α(κ) is greater than one as soon as the dimensionality ratio κ is positive. Hence, the LLR is larger than classically assumed. For instance, when κ = 0.3, α(κ) ≈ 1.5. In general, we show how to compute the scaling factor by solving a nonlinear system of two equations with two unknowns. Our mathematical arguments are involved and use techniques from approximate message passing theory, from non-asymptotic random matrix theory and from convex geometry. We also complement our mathematical study by showing that the new limiting distribution is accurate for finite sample sizes.Finally, all the results from this paper extend to some other regression models such as the probit regression model.

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

“…On the other hand, our analysis framework is based on a leave-one-out perturbation argument. This technique has been widely used to analyze high-dimensional problems with random designs, including but not limited to robust M-estimation [44,45], statistical inference for sparse regression [62], likelihood ratio test in logistic regression [108], phase synchronization [1,133], ranking from pairwise comparisons [30], community recovery [1], and covariance sketching [79]. In particular, this technique results in tight performance guarantees for the generalized power method [133], the spectral method [1,30], and convex programming approaches [30,44,108,133]; however, it has not been applied to analyze nonconvex optimization algorithms.…”

“…108) holds with probability at least 1 − O(mn −10 ) as long as m n log n. The 2 bound on x 0 − x 0,(l) …”

Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval

“…Finally, the leave-one-out arguments have been invoked to analyze other high-dimensional statistical inference problems including robust M-estimators [EKBB + 13, EK15], and maximum likelihood theory for logistic regression [SCC18], etc. In addition, [ZB17, CFMW17, AFWZ17] made use of the leave-one-out trick to derive entrywise perturbation bounds for eigenvectors resulting from certain spectral methods.…”

The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression

“…Then Cover shows that as p and n grow large in such a way that p/n → κ, the data points asymptotically overlap-with probability tending to one-if κ < 1/2 whereas they are separated-also with probability tending to one-if κ > 1/2. In the former case where the MLE exists, [17] refined Cover's result by calculating the limiting distribution of the MLE when the features x i are Gaussian.…”

“…Hence, the results from [5,6] and [17] describe a phase transition in the existence of the MLE as the dimensionality parameter κ = p/n varies around the value 1/2. Therefore, a natural question is this:…”