A Theory of the Inductive Bias and Generalization of Kernel Regression and Wide Neural Networks

Simon, James B.; Dickens, Madeline; DeWeese, Michael R.

doi:10.48550/arxiv.2110.03922

Cited by 3 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4B), generalization performance depends on values of the kernel function evaluated across this entire range of overlaps. In particular, methods from the theory of kernel regression (Sollich, 1998; Jacot et al, 2018; Bordelon et al, 2020; Canatar et al, 2021b; Simon et al, 2021) quantify a network’s performance on a learning task by decomposing the target function into a set of basis functions (Fig. 4D).…”

Section: Resultsmentioning

confidence: 99%

“…where C 1 and C 2 do not depend on α (Canatar et al, 2021b;Simon et al, 2021; see Methods). Equation ( 4) illustrates that for equal values of c α , modes with greater λ α contribute less to the generalization error.…”

Section: Geometry Of the Expansion Layer Representationmentioning

confidence: 99%

“…Standard error of the mean is computed across 10 realizations of network weights and tasks. Simon et al, 2021) quantify a network's performance on a learning task by decomposing the target function into a set of basis functions (Fig. 4D).…”

Section: Geometry Of the Expansion Layer Representationmentioning

confidence: 99%

“…To explain this result, we develop a theory that predicts the performance of cerebellum-like circuits for arbitrary learning tasks. The theory describes how properties of cerebellar architecture and activity control these networks’ inductive bias: the tendency of a network toward learning particular types of input-output mappings (Sollich, 1998; Jacot et al, 2018; Bordelon et al, 2020; Canatar et al, 2021b; Simon et al, 2021). The theory shows that inductive bias, rather than the dimension of the representation alone, is necessary to explain learning performance across tasks.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Task-dependent optimal representations for cerebellar learning

Xie

Muscinelli

Harris

et al. 2022

Preprint

View full text Add to dashboard Cite

The cerebellar granule cell layer has inspired numerous theoretical models of neural representations that support learned behaviors, beginning with the work of Marr and Albus. In these models, granule cells form a sparse, combinatorial encoding of diverse sensorimotor inputs. Such sparse representations are optimal for learning to discriminate random stimuli. However, recent observations of dense, low-dimensional activity across granule cells have called into question the role of sparse coding in these neurons. Here, we generalize theories of cerebellar learning to determine the optimal granule cell representation for tasks beyond random stimulus discrimination, including continuous input-output transformations as required for smooth motor control. We show that for such tasks, the optimal granule cell representation is substantially denser than predicted by classic theories. Our results provide a general theory of learning in cerebellum-like systems and suggest that optimal cerebellar representations are task-dependent.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Geometry Of the Expansion Layer Representationmentioning

confidence: 99%

Section: Geometry Of the Expansion Layer Representationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Task-dependent optimal representations for cerebellar learning

Xie

Muscinelli

Harris

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We qualify the performance of kernels based on the theory of the kernel method [17] and the neural tangent kernel theory for linear models [15,[30][31][32][33][34][35][36][37]. Better kernels will have flatter eigenspectra with more non-trivial kernel eigenvalues, which will lead to faster convergence speed [37] (see discussions in SM) and less generalization error for good enough alignments [38][39][40][41][42]. These features are observable through the numerical results in the kernel trick and the gradient descent dynamics.…”

mentioning

confidence: 99%

Quantum Kerr Learning

Liu¹,

Zhong²,

Otten³

et al. 2022

Preprint

View full text Add to dashboard Cite

Quantum machine learning is a rapidly evolving area that could facilitate important applications for quantum computing and significantly impact data science. In our work, we argue that a single Kerr mode might provide some extra quantum enhancements when using quantum kernel methods based on various reasons from complexity theory and physics. Furthermore, we establish an experimental protocol, which we call quantum Kerr learning based on circuit QED. A detailed study using the kernel method, neural tangent kernel theory, first-order perturbation theory of the Kerr non-linearity, and non-perturbative numerical simulations, shows quantum enhancements could happen in terms of the convergence time and the generalization error, while explicit protocols are also constructed for higher-dimensional input data.

show abstract

Quantum Kerr learning

Liu

Zhong

Otten

et al. 2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Quantum machine learning is a rapidly evolving field of research that could facilitate important applications for quantum computing and also significantly impact data-driven sciences. In our work, based on various arguments from complexity theory and physics, we demonstrate that a single Kerr mode can provide some ``quantum enhancements'' when dealing with kernel-based methods. Using kernel properties, neural tangent kernel theory, first-order perturbation theory of the Kerr non-linearity, and non-perturbative numerical simulations, we show that quantum enhancements could happen in terms of convergence time and generalization error. Furthermore, we make explicit indications on how higher-dimensional input data could be considered. Finally, we propose an experimental protocol, that we call \emph{quantum Kerr learning}, based on circuit QED.

show abstract

A Theory of the Inductive Bias and Generalization of Kernel Regression and Wide Neural Networks

Cited by 3 publications

References 0 publications

Task-dependent optimal representations for cerebellar learning

Task-dependent optimal representations for cerebellar learning

Quantum Kerr Learning

Quantum Kerr learning

Contact Info

Product

Resources

About