A Solvable Model of Neural Scaling Laws

Maloney, Alexander; Roberts, Daniel A.; Sully, James

doi:10.48550/arxiv.2210.16859

Cited by 6 publications

(11 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2021), Michaud et al (2023) or Debowski (2023), although we focus on practical models with finite capacity. The discrete nature of tokens contrasts with other recent works on scaling laws that have focused on continuous Gaussian inputs (e.g., Bahri et al, 2021;Maloney et al, 2022;Sorscher et al, 2022).…”

Section: Introductionmentioning

confidence: 89%

On Minimal Variations for Unsupervised Representation Learning

Cabannes¹,

Bietti²,

Balestriero³

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.

show abstract

Section: Introductionmentioning

confidence: 89%

On Minimal Variations for Unsupervised Representation Learning

Cabannes¹,

Bietti²,

Balestriero³

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Scaling Exponents are Task-Dependent at Late Training Time, but not at Early Time. Prior works (Dyer & Gur-Ari, 2020;Atanasov et al, 2023;Roberts et al, 2022; predict early-time finite-width loss corrections that go as 1/width near the infinite width limit in either lazy or feature-learning regimes. Bahri et al (2021) et al provide experiments demonstrating the 1/width convergence.…”

Section: Preprintmentioning

confidence: 98%

Population codes enable learning from few examples by shaping inductive bias

Bordelon

Pehlevan

2021

Preprint

View full text Add to dashboard Cite

The brain can learn from a limited number of experiences, an ability which requires suitable built in assumptions about the nature of the tasks which must be learned, or inductive biases. While inductive biases are central components of intelligence, how they are reflected in and shaped by population codes are not well-understood. To address this question, we consider biologically-plausible reading out of an arbitrary stimulus-response pattern from an arbitrary population code, and develop an analytical theory that predicts the generalization error of the readout as a function of the number of samples. We find that learning performance is controlled by the eigenspectrum of the population code's inner-product kernel, which measures the similarity of neural responses to two different input stimuli. Many different codes can realize the same kernel; by analyzing recordings from the mouse primary visual cortex, we demonstrate that biological codes are metabolically more efficient than other codes with identical kernels. We demonstrate that the spectral properties of the kernel introduce an inductive bias toward explaining stimulus-response samples with simple functions and determine compatibility of the population code with learning task, and hence the sample-efficiency of learning. While the tail of the spectrum is important for large sample size behavior of learning, for small sample sizes, the bulk of the spectrum governs generalization. We apply our theory to experimental recordings of mouse primary visual cortex neural responses, elucidating a bias towards sample-efficient learning of low frequency orientation discrimination tasks. We demonstrate this emergence of this bias in a simple model of primary visual cortex, and further show how invariances in the code to stimulus variations affect learning performance. Finally, we demonstrate that our methods are applicable to time-dependent neural codes. Overall, our study suggests sample-efficient learning as a general normative coding principle.

show abstract

“…The model of Sharma and Kaplan (2022) was also generalized by Bahri et al (2021) to account for power law scaling in training data and who additionally relate scaling exponents to a power law spectrum of certain kernels. Maloney et al (2022) develop a random-feature model of scaling, in which power law scaling comes from power law spectra of the data feature-feature covariance matrix, and scaling exponents are determined by the power law exponent over these spectra. Hutter (2021) propose a toy model of data scaling in which features are learned based on whether they've been seen during training, and a Zipfian distribution over features produces power law data scaling.…”

Section: Data Scaling (Single-epoch)mentioning

confidence: 99%

The Quantization Model of Neural Scaling

J.¹,

Liu²,

Girit³

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose the Quantization Model of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and also the sudden emergence of new capabilities with scale. We derive this model from what we call the Quantization Hypothesis, where learned network capabilities are quantized into discrete chunks (quanta). We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss. We validate this prediction on toy datasets, then study how scaling curves decompose for large language models. Using language model internals, we auto-discover diverse model capabilities (quanta) and find tentative evidence that the distribution over corresponding subproblems in the prediction of natural text is compatible with the power law predicted from the neural scaling exponent as predicted from our theory.

show abstract

A Solvable Model of Neural Scaling Laws

Cited by 6 publications

References 60 publications

On Minimal Variations for Unsupervised Representation Learning

On Minimal Variations for Unsupervised Representation Learning

Population codes enable learning from few examples by shaping inductive bias

The Quantization Model of Neural Scaling

Contact Info

Product

Resources

About