Dynamics of Learning in Multilayer Perceptrons Near Singularities

Abstract

The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the "plateau." It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the "blow-down" technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods.

Learning Dynamics of the Complex-Valued Neural Network in the Neighborhood of Singular Points

“…Recently, it has turned out that the singularity has a negative effect on learning dynamics in the real-valued neural networks [12,13,15]. That is, the hierarchical structure or a symmetric property on exchange of weights of the the real-valued neural networks have singular points.…”

“…On one hand, the researches on the singularity of the learning machines with a hierarchical structure have progressed in the past several years [12][13][14][15]. It has turned out that the singularity has a negative effect on learning dynamics in the learning machines such as real-valued neural networks and gaussian mixture models.…”

Adaptive stimulus optimization for sensory systems neuroscience

“…Although a generalized linear model allows limited non-linearities, it enjoys tractable and consistent estimation procedures without problems of local minima (Paninski, 2004). Identifying more complex non-linear models like hierarchical neural networks from physiological data tends to be harder due to problems like local minima and plateaus in the error surface (Amari et al, 2006; Cousseau et al, 2008; Wei and Amari, 2008; Wei et al, 2008). …”

Online Learning Dynamics of Radial Basis Function Neural Networks near the Singularity

“…In a recent article, Amari, Park and Ozeki [1] argued that the singular structure is ubiquitous, covering a wide range of models from MLPs, Gaussian mixtures, ARMA time-series and many others, and the learning dynamics will be heavily influenced by the singularities. The special case of MLPs with unidentifiable parameters has been discussed both numerically [12] and analytically [4], and some interesting behaviours have been revealed in these studies.…”