Tuning and jamming reduced to their minima

Ruiz-García, Miguel; Liu, Andrea J.; Katifori, Eleni

doi:10.1103/physreve.100.052608

Cited by 17 publications

(14 citation statements)

References 32 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most notably, [16,17] show that curriculum can lead to faster learning in a simple setting, but the effects of curriculum on asymptotic generalisation and the dependence on task structure remain unclear. A hint that indeed curriculum learning might lead to statistically different minima comes from a connection between constraint-satisfaction problems and physics results on flow networks [18], but to our knowledge no direct result has been reported in the modern theoretical machine learning literature.…”

Section: Introductionmentioning

confidence: 91%

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Saglietti¹,

Mannelli²,

Saxe³

2021

Preprint

View full text Add to dashboard Cite

In humans and animals, curriculum learning-presenting data in a curated order-is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not widely used and empirically often yield only moderate benefits. This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods. Curricula could in principle change both the learning speed and asymptotic performance of a model. To study the former, we provide an exact description of the online learning setting, confirming the long-standing experimental observation that curricula can modestly speed up learning. To study the latter, we derive performance in a batch learning setting, in which a network trains to convergence in successive phases of learning on dataset slices of varying difficulty. With standard training losses, curriculum does not provide generalisation benefit, in line with empirical observations. However, we show that by connecting different learning phases through simple Gaussian priors, curriculum can yield a large improvement in test performance. Taken together, our reduced analytical descriptions help reconcile apparently conflicting empirical results and trace regimes where curriculum learning yields the largest gains. More broadly, our results suggest that fully exploiting a curriculum may require explicit changes to the loss function at curriculum boundaries.

show abstract

Section: Introductionmentioning

confidence: 91%

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Saglietti¹,

Mannelli²,

Saxe³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Depending on the values of Γ i , the topography of the loss function will change, but the loss function will still vanish at the same global minima, which are unaffected by the value of Γ i . This transformation was motivated by recent work in which the topography of the loss function was changed to improve the tuning of physical flow networks [39]. Here, we use Γ i to emphasize one class relative to the others for a period T , and cycle through all the classes in turn so the total duration of a cycle that passes through all classes is CT .…”

Section: Myrtle5 and Cifar10 Phase Diagramsmentioning

confidence: 99%

Tilting the playing field: Dynamical loss functions for machine learning

Ruiz-Garcia,

Zhang,

Schoenholz

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamical loss functions can lead to successful training for networks that fail to find a deep minima of the standard cross-entropy loss. In overparameterized networks, dynamical loss functions can lead to better generalization. Improvement arises from the interplay of the changing loss landscape with the dynamics of the system as it evolves to minimize the loss. In particular, as the loss function oscillates, instabilities develop in the form of bifurcation cascades, which we study using the Hessian and Neural Tangent Kernel. Valleys in the landscape widen and deepen, and then narrow and rise as the loss landscape changes during a cycle. As the landscape narrows, the learning rate becomes too large and the network becomes unstable and bounces around the valley. This process ultimately pushes the system into deeper and wider regions of the loss landscape and is characterized by decreasing eigenvalues of the Hessian. This results in better regularized models with improved generalization performance.

show abstract

“…Here we follow a different approach to learning that exploits physical processes, involving simple and local rules, in lieu of complex ones inspired by neurons or non-local computer science algorithms. In previous work, simulated and laboratory mechanical networks, and simulated flow networks, have been trained to perform desired tasks by adjusting their internal degrees of freedom [26][27][28][29][30][31][32][33][34][35][36][37][38][39]. This has been accomplished either by minimizing a global cost function [26][27][28][29][30] or using local rules [31][32][33][34][35][36][37][38][39], in which each edge of the network adjusts some property -such as its mechanical stiffness -that we will refer to as a 'learning degree of freedom' in response to the stress on it.…”

Section: Introductionmentioning

confidence: 99%

Demonstration of Decentralized, Physics-Driven Learning

Dillavou,

Stern,

Liu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The brain is a physical system capable of producing immensely complex collective properties on demand. Unlike a typical man-made computer, which is limited by the bottleneck of one or more central processors, self-adjusting elements of the brain (neurons and synapses) operate entirely in parallel. Here we report laboratory demonstration of a new kind of physics-driven learning network comprised of identical self-adjusting variable resistors. Our system is a realization of coupled learning, a recently-introduced theoretical framework specifying properties that enable adaptive learning in physical networks. The inputs are electrical stimuli and physics performs the output 'computations' through the natural tendency to minimize energy dissipation across the system. Thus the outputs are analog physical responses to the input stimuli, rather than digital computations of a central processor. When exposed to training data, each edge of the network adjusts its own resistance in parallel using only local information, effectively training itself to generate the desired output. Our physical learning machine learns a range of tasks, including regression and classification, and switches between them on demand. It operates using well-understood physical principles and is made using standard, commercially available electronic components. As a result our system is massively scalable and amenable to theoretical understanding, combining the simplicity of artificial neural networks with the distributed parallel computation and robustness to damage boasted by biological neuron networks.

show abstract

Tuning and jamming reduced to their minima

Cited by 17 publications

References 32 publications

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Tilting the playing field: Dynamical loss functions for machine learning

Demonstration of Decentralized, Physics-Driven Learning

Contact Info

Product

Resources

About