Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron

Satoh, Seiya; Nakano, Ryoko

doi:10.1007/s11063-013-9283-z

Cited by 16 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [175], the proposed online constructive learning algorithm for the MLP integrates the weight scaling technique [176,177] for escaping local minima, and the quadratic programming (QP) and linear programming (LP)-based procedure for initializing the weights of a newly added neuron. In [178], the proposed constructive approach for MLP learning finds excellent solutions by exploiting the flat regions.…”

Section: Constructive Approach: Network Growingmentioning

confidence: 99%

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Leung

Mow

et al. 2022

Mathematics

View full text Add to dashboard Cite

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

show abstract

Section: Constructive Approach: Network Growingmentioning

confidence: 99%

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Leung

Mow

et al. 2022

Mathematics

View full text Add to dashboard Cite

show abstract

“…The proposed method, a complex version of SSF [13], is explained. C-SSF starts search from C-MLP(J=1) and then gradually increases J one by one until J max .…”

Section: C-ssf: Complex Singularity Stairs Followingmentioning

confidence: 99%

“…Singular regions have been avoided as far as possible [1] because they cause serious stagnation of learning; however, we think they can be utilized as excellent initial points for successive search. In this viewpoint, a method called SSF (Singularity Stairs Following) [9,13] was once proposed, which makes good use of singular regions to stably and successively find excellent solutions for real-valued MLPs. This paper proposes a complex version of SSF, called Complex Singularity Stairs Following (C-SSF).…”

Section: Introductionmentioning

confidence: 99%

Complex-Valued Multilayer Perceptron Search Utilizing Singular Regions of Complex-Valued Parameter Space

Satoh

Nakano

2014

Artificial Neural Networks and Machine Learning – ICANN 2014

Self Cite

View full text Add to dashboard Cite

Abstract. In the search space of a complex-valued multilayer perceptron having J hidden units, C-MLP(J), there exist flat areas called singular regions, as is the case with a real-valued MLP. The singular regions cause serious stagnation of learning, preventing usual search methods from finding an excellent solution. However, there exist descending paths from the regions since most points in the regions are saddles. This paper proposes a completely new learning method that does not avoid but makes good use of singular regions to stably and successively find excellent solutions commensurate with C-MLP(J). Our experiments showed the proposed method worked well.

show abstract

“…Learning models have been studied in relation to singular points (Amari, Park, & Ozeki, 2006;Wei, Zhang, Cousseau, Ozeki, & Amari, 2008;Cousseaeu, Ozeki, & Amari, 2008;Satoh & Nakano, 2013;Nitta, 2013a). For example, learning models with hierarchical structures or a symmetric property on exchange of weights, such as hierarchical neural networks and mixture models, mostly have singular points.…”

Section: Introductionmentioning

confidence: 99%

Learning Dynamics of a Single Polar Variable Complex-Valued Neuron

Nitta

2015

Neural Computation

View full text Add to dashboard Cite

This letter investigates the characteristics of the complex-valued neuron model with parameters represented by polar coordinates (called polar variable complex-valued neuron). The parameters of the polar variable complex-valued neuron are unidentifiable. The plateau phenomenon can occur during learning of the polar variable complex-valued neuron. Furthermore, computer simulations suggest that a single polar variable complex-valued neuron has the following characteristics in the case of using the steepest gradient-descent method with square error: (1) unidentifiable parameters (singular points) degrade the learning speed and (2) a plateau can occur during learning. When the weight is attracted to the singular point, the learning tends to become stuck. However, computer simulations also show that the steepest gradient-descent method with amplitude-phase error and the complex-valued natural gradient method could reduce the effects of the singular points. The learning dynamics near singular points depends on the error functions and the training algorithms used.

show abstract

Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron

Cited by 16 publications

References 11 publications

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Complex-Valued Multilayer Perceptron Search Utilizing Singular Regions of Complex-Valued Parameter Space

Learning Dynamics of a Single Polar Variable Complex-Valued Neuron

Contact Info

Product

Resources

About