Optimal errors and phase transitions in high-dimensional generalized linear models

Abstract

Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or “free entropy”) from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms.

Inference With Deep Generative Priors in High Dimensions

“…• Since the original conference versions of this paper [1], [2], formulae for the minimum mean-squared error (MMSE) for inference in deep networks have been conjectured in [47]- [49]. As discussed in Section IV-C, these formulae are based on heuristic techniques, such as the replica method from statistical physics, and have been rigorously proven in special cases [50].…”

Annealing and Replica-Symmetry in Deep Boltzmann Machines

“…The duality among disorder-to-order transition in statistical mechanics of disordered systems and detectability-undetectability transition in machine learning (see e.g. [6,8,9,10,12,15,19]) suggests that the knowledge of the optimal shapes stemmed from the former could play some role in the latter.…”

Concentration of the Matrix-Valued Minimum Mean-Square Error in Optimal Bayesian Inference

“…The fluctuations of this matrix are easier to control than the ones of the overlap because L is related to the λ ngradient of the free energy, which is self-averaging by hypothesis (9). The proof is a straightforward extension to the matrix case of the one found in [23], [30] and requires no new ideas. This general result does not depend on the fact that we consider optimal Bayesian inference; it is only a consequence of the perturbation, i.e., the side information coming from the channel (6).…”

“…the product posterior measure, and g is any bounded function. This innocent-looking key identity on which relies the whole proof follows directly from Bayes' law -thus the importance of placing ourselves in the Bayesian optimal setting-, see [23], [30]. Applied to…”