Phase transitions of neural networks

Kinzel, Wolfgang

doi:10.1080/014186398258861

Cited by 6 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A common phenomenon observed in the studies of learning from examples is the existence of phase transitions with abrupt improvement in the generalization ability of the networks once the training examples are sufficiently numerous, or the global parameters (e.g. the weight decay) is suitably tuned [23][24][25][26][27]. These transitions are often discontinuous.…”

Section: Introductionmentioning

confidence: 99%

Cavity approach to noisy learning in nonlinear perceptrons

Luo

Wong

2001

Phys. Rev. E

View full text Add to dashboard Cite

We analyze the learning of noisy teacher-generated examples by nonlinear and differentiable student perceptrons using the cavity method. The generic activation of an example is a function of the cavity activation of the example, which is its activation in the perceptron that learns without the example. Mean-field equations for the macroscopic parameters and the stability condition yield results consistent with the replica method. When a single value of the cavity activation maps to multiple values of the generic activation, there is a competition in learning strategy between preferentially learning an example and sacrificing it in favor of the background adjustment. We find parameter regimes in which examples are learned preferentially or sacrificially, leading to a gap in the activation distribution. Full phase diagrams of this complex system are presented, and the theory predicts the existence of a phase transition from poor to good generalization states in the system. Simulation results confirm the theoretical predictions.

show abstract

Section: Introductionmentioning

confidence: 99%

Cavity approach to noisy learning in nonlinear perceptrons

Luo

Wong

2001

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…See e.g. [2,3,4] for reviews and [5] for a discussion focussed on phase transitions in neural networks.…”

mentioning

confidence: 99%

Phase transitions in soft-committee machines

Biehl¹,

Ahr²,

Schlösser³

1999

Computer Physics Communications

View full text Add to dashboard Cite

Equilibrium statistical physics is applied to layered neural networks with differentiable activation functions. A first analysis of off-line learning in soft-committee machines with a finite number (K) of hidden units learning a perfectly matching rule is performed. Our results are exact in the limit of high training temperatures (β → 0). For K = 2 we find a second order phase transition from unspecialized to specialized student configurations at a critical size P of the training set, whereas for K ≥ 3 the transition is first order. Monte Carlo simulations indicate that our results are also valid for moderately low temperatures qualitatively. The limit K → ∞ can be performed analytically, the transition occurs after presenting on the order of N K/β examples. However, an unspecialized metastable state persists up to P ∝ N K 2 /β.

show abstract

“…The application of methods of statistical mechanics to neural networks has yielded a wealth of results. Analysing the prototype architecture, the perceptron produces several types of behaviour such as phase transitions [1][2][3]. Here a learning algorithm for the one-layer network in the case of Ising weights is presented.…”

Section: Introductionmentioning

confidence: 99%

On-line learning in the Ising perceptron

Rosen-Zvi¹

2000

J. Phys. A: Math. Gen.

View full text Add to dashboard Cite

On-line learning of both binary and continuous rules in an Ising space is studied. Learning is achieved by using an artificial parameter, a weight vector J , which is constrained to the surface of a hypersphere (spherical constraint). In the case of a binary rule the generalization error decays to zero super-exponentially as exp(−Cα 2), where α is the number of examples divided by N, the size of the input vector, and C > 0. Much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp(−e |λ|α). The number of steps required for perfect learning is estimated for both scenarios and compared with simulations.

show abstract

Phase transitions of neural networks

Cited by 6 publications

References 18 publications

Cavity approach to noisy learning in nonlinear perceptrons

Cavity approach to noisy learning in nonlinear perceptrons

Phase transitions in soft-committee machines

On-line learning in the Ising perceptron

Contact Info

Product

Resources

About