Universality laws for Gaussian mixtures in generalized linear models*

Abstract

Let ( x i , y i ) i = 1 , … , n denote independent samples from a general mixture distribution ∑ c ∈ C ρ c P c x , and consider the hypothesis class of generalized linear models y ^ = F ( Θ ⊤ x ) . In this study, we investigate the asymptotic joint statistics of a family of generalized linear estimators ( Θ 1 , … , Θ M ) obtained either from (a) minimizing an empirical risk R ^ n ( Θ ; X , y ) or (b) sampling from the associated Gibbs measure exp ⁡ ( − β n R ^ n ( Θ ; X , y ) ) . Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (in a weak sense) only on the means and covariances of the class conditional features distribution P c x . In particular, this allows us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results in different machine learning tasks of interest, such as ensembling and uncertainty quantification.