Gaussian Universality of Linear Classifiers with Random Labels in High-Dimension

Gerace, Federica; Krząkała, Florent; Loureiro, Bruno; Stephan, Ludovic; Zdeborová, Lenka

doi:10.48550/arxiv.2205.13303

Cited by 4 publications

(4 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies assessed the learning performance of cerebellum-like systems with a model Purkinje cell that associates random patterns of mossy fiber activity with one of two randomly assigned categories (Marr, 1969; Albus, 1971; Brunel et al, 2004; Babadi and Sompolinsky, 2014; Litwin-Kumar et al, 2017; Cayco-Gajic et al, 2017), a common benchmark for artificial learning systems (Gerace et al, 2022). In this case, a low coding level increases the dimension of the granule cell representation, permitting more associations to be stored.…”

Section: Discussionmentioning

confidence: 99%

Task-dependent optimal representations for cerebellar learning

Xie

Muscinelli

Harris

et al. 2022

Preprint

View full text Add to dashboard Cite

The cerebellar granule cell layer has inspired numerous theoretical models of neural representations that support learned behaviors, beginning with the work of Marr and Albus. In these models, granule cells form a sparse, combinatorial encoding of diverse sensorimotor inputs. Such sparse representations are optimal for learning to discriminate random stimuli. However, recent observations of dense, low-dimensional activity across granule cells have called into question the role of sparse coding in these neurons. Here, we generalize theories of cerebellar learning to determine the optimal granule cell representation for tasks beyond random stimulus discrimination, including continuous input-output transformations as required for smooth motor control. We show that for such tasks, the optimal granule cell representation is substantially denser than predicted by classic theories. Our results provide a general theory of learning in cerebellum-like systems and suggest that optimal cerebellar representations are task-dependent.

show abstract

Section: Discussionmentioning

confidence: 99%

Task-dependent optimal representations for cerebellar learning

Xie

Muscinelli

Harris

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, this setting is not as restrictive as it appears. In fact, recent studies have shown Gaussian universality for GLMs with a large class of features, wherein their asymptotic behaviors can be computed to leading order under a simpler model with Gaussian features [65,[75][76][77]. Utilizing this correspondence, the statistical physics-based method is expected to be a valuable tool for analyzing the statistical methods to find limitation in the conventional statistical theory, analyze the α ∼ O(1) region, and construct efficient algorithms.…”

Section: Discussionmentioning

confidence: 99%

Prediction errors for penalized regressions based on generalized approximate message passing

Sakata

2023

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

We discuss the prediction accuracy of assumed statistical models in terms of prediction errors for the generalized linear model and penalized maximum likelihood methods. We derive the forms of estimators for the prediction errors: $C_p$ criterion, information criteria, and leave-one-out cross validation (LOOCV) error, using the generalized approximate message passing (GAMP) algorithm and replica method. These estimators coincide with each other when the number of model parameters is sufficiently small; however, there is a discrepancy between them in particular in the parameter region where the number of model parameters is larger than the data dimension. In this paper, we review the prediction errors and corresponding estimators, and discuss their differences. In the framework of GAMP, we show that the information criteria can be expressed by using the variance of the estimates. Further, we demonstrate how to approach LOOCV error from the information criteria by utilizing the expression provided by GAMP.

show abstract

“…to be robust and hold for a larger class of random matrices. Rigorous proofs are presented in [65][66][67][68][69], where the asymptotic prediction is shown to have a universal limit (as p → ∞) with respect to random matrices with i.i.d. entries.…”

Section: Synthetic Data: Universality Of the Gaussian Designmentioning

confidence: 99%

Generalized Penalized Constrained Regression: Sharp Guarantees in High Dimensions with Noisy Features

Alrashdi,

Alazmi,

Alrasheedi

2023

Mathematics

View full text Add to dashboard Cite

The generalized penalized constrained regression (G-PCR) is a penalized model for high-dimensional linear inverse problems with structured features. This paper presents a sharp error performance analysis of the G-PCR in the over-parameterized high-dimensional setting. The analysis is carried out under the assumption of a noisy or erroneous Gaussian features matrix. To assess the performance of the G-PCR problem, the study employs multiple metrics such as prediction risk, cosine similarity, and the probabilities of misdetection and false alarm. These metrics offer valuable insights into the accuracy and reliability of the G-PCR model under different circumstances. Furthermore, the derived results are specialized and applied to well-known instances of G-PCR, including l1-norm penalized regression for sparse signal recovery and l2-norm (ridge) penalization. These specific instances are widely utilized in regression analysis for purposes such as feature selection and model regularization. To validate the obtained results, the paper provides numerical simulations conducted on both real-world and synthetic datasets. Using extensive simulations, we show the universality and robustness of the results of this work to the assumed Gaussian distribution of the features matrix. We empirically investigate the so-called double descent phenomenon and show how optimal selection of the hyper-parameters of the G-PCR can help mitigate this phenomenon. The derived expressions and insights from this study can be utilized to optimally select the hyper-parameters of the G-PCR. By leveraging these findings, one can make well-informed decisions regarding the configuration and fine-tuning of the G-PCR model, taking into consideration the specific problem at hand as well as the presence of noisy features in the high-dimensional setting.

show abstract

Gaussian Universality of Linear Classifiers with Random Labels in High-Dimension

Cited by 4 publications

References 48 publications

Task-dependent optimal representations for cerebellar learning

Task-dependent optimal representations for cerebellar learning

Prediction errors for penalized regressions based on generalized approximate message passing

Generalized Penalized Constrained Regression: Sharp Guarantees in High Dimensions with Noisy Features

Contact Info

Product

Resources

About