Statistical physics and practical training of soft-committee machines

Ahr, M.; Biehl, Michael; Urbanczik, Robert

doi:10.1007/s100510050889

Cited by 14 publications

(25 citation statements)

References 24 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The term Soft Committee Machine (SCM) has been coined for feedforward neural networks with sigmoidal activations in a single hidden layer and a linear output unit (see, for instance, [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 55 , 56 ]). Its structure resembles that of a (crisp) committee machine with binary threshold hidden units, where the network’s response is given by their majority vote (see [ 5 , 6 , 7 ] and references therein).…”

Section: Models and Mathematical Analysismentioning

confidence: 99%

Statistical Mechanics of On-Line Learning Under Concept Drift

Straat

Abadi

Göpfert

et al. 2018

Entropy

View full text Add to dashboard Cite

We introduce a modeling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means of prototype-based Learning Vector Quantization (LVQ). In the second, we study the training of layered neural networks with sigmoidal activations for the purpose of regression. In both cases, the target, i.e., the classification or regression scheme, is considered to change continuously while the system is trained from a stream of labeled data. We extend and apply methods borrowed from statistical physics which have been used frequently for the exact description of training dynamics in stationary environments. Extensions of the approach allow for the computation of typical learning curves in the presence of concept drift in a variety of model situations. First results are presented and discussed for stochastic drift processes in classification and regression problems. They indicate that LVQ is capable of tracking a classification scheme under drift to a non-trivial extent. Furthermore, we show that concept drift can cause the persistence of sub-optimal plateau states in gradient based training of layered neural networks for regression.

show abstract

Section: Models and Mathematical Analysismentioning

confidence: 99%

Statistical Mechanics of On-Line Learning Under Concept Drift

Straat

Abadi

Göpfert

et al. 2018

Entropy

View full text Add to dashboard Cite

show abstract

“…Here G r is an effective Hamiltonian and the entropy term s = (1/2) ln det C, where C is the K(n + 1) × K(n + 1)-dimensional matrix of the order parameters [10]. We have further introduced the rescaled number of examples, α = P/(NK).…”

mentioning

confidence: 99%

“…The remaining order parameters Q0 , Q1 , δ 0 , δ 1 and m parametrize the distribution of overlaps between the weight vectors of different students. Note that as in [10], using the saddle point equations for the free energy, one may analytically eliminate the unspecialized order parameters R, Q, Q0 and Q1 .…”

mentioning

confidence: 99%

Noisy regression and classification with continuous multilayer networks

Ahr¹,

Biehl²,

Urbanczik³

1999

J. Phys. A: Math. Gen.

View full text Add to dashboard Cite

We investigate zero temperature Gibbs learning for two classes of unrealizable rules which play an important rôle in practical applications of multilayer neural networks with differentiable activation functions: classification problems and noisy regression problems. Considering one step of replica symmetry breaking, we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable. Further the classification problem is shown to be formally equivalent to the noisy regression problem.

show abstract

“…The term Soft Committee Machine (SCM) has been established for shallow feedforward neural networks with a single hidden layer and a linear output unit, see for instance [2,8,9,11,26,42,44,45,49]. Its structure resembles that of a (crisp) committee machine with binary threshold hidden units, where the network output is given by their majority vote, see [4,19,53] and references therein.…”

Section: Layered Neural Networkmentioning

confidence: 99%

Supervised learning in the presence of concept drift: a modelling framework

Straat

Abadi

Kan

et al. 2021

Neural Comput & Applic

Self Cite

View full text Add to dashboard Cite

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

show abstract

Statistical physics and practical training of soft-committee machines

Cited by 14 publications

References 24 publications

Statistical Mechanics of On-Line Learning Under Concept Drift

Statistical Mechanics of On-Line Learning Under Concept Drift

Noisy regression and classification with continuous multilayer networks

Supervised learning in the presence of concept drift: a modelling framework

Contact Info

Product

Resources

About