The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression

Candès, Emmanuel J.; Sur, Pragya

doi:10.1214/18-aos1789

Cited by 72 publications

(60 citation statements)

References 22 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This happens because in such cases, there is a perfect separating hyperplane—separating the cases from the controls if you will—sending the MLE to infinity. It turns out that a companion paper (39) precisely characterizes the region in which the MLE exists.…”

Section: Resultsmentioning

confidence: 99%

A modern maximum-likelihood theory for high-dimensional logistic regression

Sur

Candès

2019

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

196

185

View full text Add to dashboard Cite

Students in statistics or data science usually learn early on that when the sample size n is large relative to the number of variables p, fitting a logistic model by the method of maximum likelihood produces estimates that are consistent and that there are well-known formulas that quantify the variability of these estimates which are used for the purpose of statistical inference. We are often told that these calculations are approximately valid if we have 5 to 10 observations per unknown parameter. This paper shows that this is far from the case, and consequently, inferences produced by common software packages are often unreliable. Consider a logistic model with independent features in which n and p become increasingly large in a fixed ratio. We prove that (i) the maximum-likelihood estimate (MLE) is biased, (ii) the variability of the MLE is far greater than classically estimated, and (iii) the likelihood-ratio test (LRT) is not distributed as a χ2. The bias of the MLE yields wrong predictions for the probability of a case based on observed values of the covariates. We present a theory, which provides explicit expressions for the asymptotic bias and variance of the MLE and the asymptotic distribution of the LRT. We empirically demonstrate that these results are accurate in finite samples. Our results depend only on a single measure of signal strength, which leads to concrete proposals for obtaining accurate inference in finite samples through the estimate of this measure.

show abstract

Section: Resultsmentioning

confidence: 99%

A modern maximum-likelihood theory for high-dimensional logistic regression

Sur

Candès

2019

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

196

185

View full text Add to dashboard Cite

show abstract

“…Logistic regression is a mathematical modeling approach that can be used to describe the relationship of several variables to a dichotomous dependent variable [ 66 , 67 ]. LOGREG theory is simple and easy to understand, but it lacks robustness and accuracy when there is noise in the data [ 68 ].…”

Section: Methodsmentioning

confidence: 99%

A Labeling Method for Financial Time Series Prediction Based on Trends

Wang

et al. 2020

Entropy

View full text Add to dashboard Cite

Time series prediction has been widely applied to the finance industry in applications such as stock market price and commodity price forecasting. Machine learning methods have been widely used in financial time series prediction in recent years. How to label financial time series data to determine the prediction accuracy of machine learning models and subsequently determine final investment returns is a hot topic. Existing labeling methods of financial time series mainly label data by comparing the current data with those of a short time period in the future. However, financial time series data are typically non-linear with obvious short-term randomness. Therefore, these labeling methods have not captured the continuous trend features of financial time series data, leading to a difference between their labeling results and real market trends. In this paper, a new labeling method called “continuous trend labeling” is proposed to address the above problem. In the feature preprocessing stage, this paper proposed a new method that can avoid the problem of look-ahead bias in traditional data standardization or normalization processes. Then, a detailed logical explanation given, a definition of continuous trend labeling was proposed and also an automatic labeling algorithm was given to extract the continuous trend features of financial time series data. Experiments on the Shanghai Composite Index and Shenzhen Component Index and some stocks of China showed that our labeling method is a much better state-of-the-art labeling method in terms of classification accuracy and some other classification evaluation metrics. The results of the paper also proved that deep learning models such as LSTM and GRU are more suitable for dealing with the prediction of financial time series data.

show abstract

“…Step 1: (Setup). Our proof is inspired by Lemma 4 of Candès et al (2020). The proof is fairly straightforward and relies mainly on the convexity properties of L and the concentration statements established in Lemma I.2.…”

Section: I2 Proof For Proposition 52mentioning

confidence: 99%

Exponential-family embedding with application to cell developmental trajectories for single-cell RNA-seq data

Lin

Lei

Roeder

2020

Preprint

View full text Add to dashboard Cite

Scientists often embed cells into a lower-dimensional space when studying single-cell RNA-seq data for improved downstream analyses such as developmental trajectory analyses, but the statistical properties of such non-linear embedding methods are often not well understood. In this article, we develop the eSVD (exponential-family SVD), a non-linear embedding method for both cells and genes jointly with respect to a random dot product model using exponential-family distributions. Our estimator uses alternating minimization, which enables us to have a computationally-efficient method, prove the identifiability conditions and consistency of our method, and provide statistically-principled procedures to tune our method. All these qualities help advance the single-cell embedding literature, and we provide extensive simulations to demonstrate that the eSVD is competitive compared to other embedding methods.We apply the eSVD via Gaussian distributions where the standard deviations are proportional to the means to analyze a single-cell dataset of oligodendrocytes in mouse brains (Marques et al., 2016). Using the eSVD estimated embedding, we then investigate the cell developmental trajectories of the oligodendrocytes. While previous results are not able to distinguish the trajectories among the mature oligodendrocyte cell types, our diagnostics and results demonstrate there are two major developmental trajectories that diverge at mature oligodendrocytes.

show abstract

The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression

Cited by 72 publications

References 22 publications

A modern maximum-likelihood theory for high-dimensional logistic regression

A modern maximum-likelihood theory for high-dimensional logistic regression

A Labeling Method for Financial Time Series Prediction Based on Trends

Exponential-family embedding with application to cell developmental trajectories for single-cell RNA-seq data

Contact Info

Product

Resources

About