Noisy regression and classification with continuous multilayer networks

Abstract

We investigate zero temperature Gibbs learning for two classes of unrealizable rules which play an important rôle in practical applications of multilayer neural networks with differentiable activation functions: classification problems and noisy regression problems. Considering one step of replica symmetry breaking, we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable. Further the classification problem is shown to be formally equivalent to the noisy regression problem.

Statistical physics and practical training of soft-committee machines

“…Note that several such states can be approached successively while learning with a fixed rate η. Details of the simulations will be explained in a forthcoming publication [22]. Simulation results are in good aggreement with the theoretical analysis for a range of finite η.…”

“…For threshold activation functions it was observed in [19] that this transition is affected by replica symmetry breaking, resulting in a lower critical value for α than predicted in replica symmetry and changing the nature of the transition from first to second order. A more detailed discussion of this transition for the present case will be given elsewhere [22].…”

“…Note that several such states can be approached successively while learning with a fixed rate η. Details of the simulations will be explained in a forthcoming publication [22]. Left panel: Solid lines show ǫ g vs. ǫ t as obtained from the Gibbs ensemble by eliminating β and disregarding stability criteria.…”

Weight-decay induced phase transitions in multilayer neural networks

“…Meanwhile these studies are being extended to the more application relevant scenario of networks with continuous activation function and output. [11,12,13] The soft-committee machine is a two-layered neural network which consists of a layer of K hidden units, all of which are connected with the entire N-dimensional input ξ. The total output σ ist proportional to the sum of outputs of all hidden units:…”

“…the error function. Networks of this type have been studied in the limit of high temperature [11], the annealed approximation [13], and by means of the replica formalism [12]. All these studies imposed the simplifying condition that the order parameters Q ij = J i • J j /N are restricted to the value 1 for i = j, so the length of the student vectors is fixed to that of the teacher vectors.…”