Phase transitions in soft-committee machines

Biehl, Michael; Ahr, M.; Schlösser, E.

doi:10.1016/s0010-4655(06)70021-2

Cited by 5 publications

(24 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The term Soft Committee Machine (SCM) has been coined for feedforward neural networks with sigmoidal activations in a single hidden layer and a linear output unit (see, for instance, [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 55 , 56 ]). Its structure resembles that of a (crisp) committee machine with binary threshold hidden units, where the network’s response is given by their majority vote (see [ 5 , 6 , 7 ] and references therein).…”

Section: Models and Mathematical Analysismentioning

confidence: 99%

Statistical Mechanics of On-Line Learning Under Concept Drift

Straat

Abadi

Göpfert

et al. 2018

Entropy

Self Cite

View full text Add to dashboard Cite

We introduce a modeling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means of prototype-based Learning Vector Quantization (LVQ). In the second, we study the training of layered neural networks with sigmoidal activations for the purpose of regression. In both cases, the target, i.e., the classification or regression scheme, is considered to change continuously while the system is trained from a stream of labeled data. We extend and apply methods borrowed from statistical physics which have been used frequently for the exact description of training dynamics in stationary environments. Extensions of the approach allow for the computation of typical learning curves in the presence of concept drift in a variety of model situations. First results are presented and discussed for stochastic drift processes in classification and regression problems. They indicate that LVQ is capable of tracking a classification scheme under drift to a non-trivial extent. Furthermore, we show that concept drift can cause the persistence of sub-optimal plateau states in gradient based training of layered neural networks for regression.

show abstract

Section: Models and Mathematical Analysismentioning

confidence: 99%

Statistical Mechanics of On-Line Learning Under Concept Drift

Straat

Abadi

Göpfert

et al. 2018

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…The quantity e N s corresponds to the volume in weight space that is consistent with a given configuration of order parameters. Independent of the activation functions or other details of the learning problem, one obtains for large N [30], [31]…”

Section: Thermal Equilibrium and The High-temperature Limitmentioning

confidence: 99%

“…Omitting additive constants and assuming the normalization (6) and site-symmetry (10), the entropy term reads [30], [31]…”

Section: Thermal Equilibrium and The High-temperature Limitmentioning

confidence: 99%

See 1 more Smart Citation

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Oostwal,

Straat,

Biehl

2019

Preprint

Self Cite

View full text Add to dashboard Cite

We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios. The systems exhibit sudden changes of the generalization performance via the process of hidden unit specialization at critical sizes of the training set. Surprisingly, our results show that the training behavior of ReLU networks is qualitatively different from that of networks with sigmoidal activations. In networks with K ≥ 3 sigmoidal hidden units, the transition is discontinuous: Specialized network configurations co-exist and compete with states of poor performance even for very large training sets. On the contrary, the use of ReLU activations results in continuous transitions for all K. For large enough training sets, two competing, differently specialized states display similar generalization abilities, which coincide exactly for large networks in the limit K → ∞. * (who contributed equally)This work was supported through the Northern Netherlands Region of Smart Factories (RoSF) consortium (see: http://www.rosf.nl).E. Oostwal, M. Straat, and M. Biehl are with the Bernoulli Institute for Mathematics,

show abstract

“…The research field of deep learning has recently attracted considerable attention due to significant progress in performing tasks relevant to many different applications [1][2][3][4][5]. Neural networks are learning machines inspired by the structure of the human brain [6], which have been studied with methods from statistical mechanics [5,[7][8][9][10][11], starting with simpler versions such as the perceptron [12][13][14] and also including two-layer networks [15][16][17][18][19][20][21][22][23][24][25][26]. Often, learning is studied in the framework of the student-teacher scenario, in which a student has to learn the connection vectors according to which a teacher classifies input patterns [10].…”

mentioning

confidence: 99%

Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

Richert,

Worschech,

Rosenow

2021

Preprint

View full text Add to dashboard Cite

Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.

show abstract

Phase transitions in soft-committee machines

Cited by 5 publications

References 16 publications

Statistical Mechanics of On-Line Learning Under Concept Drift

Statistical Mechanics of On-Line Learning Under Concept Drift

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

Contact Info

Product

Resources

About