On structure-exploiting trust-region regularized nonlinear least squares algorithms for neural-network learning

Mizutani, Eiji; Demmel, James

doi:10.1016/s0893-6080(03)00085-6

Cited by 24 publications

(21 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In consequence, the use of the global Hessian would reduce the number of epochs required to achieve a target error level, although it would increase the time per epoch. (This is confirmed with the Gauss-Newton Hessian case in our previous work [18], [11] using the trust-region method [10], [12]). The posed tradeoff between the global and local Hessian matrices leads to a delicate balancing act to be pursued further.…”

Section: Discussionsupporting

confidence: 83%

See 1 more Smart Citation

A block-approximate local Hessian-matrix analysis for CANFIS neuro-fuzzy modular network learning

Mizutani

2012

2012 IEEE International Conference on Fuzzy Systems

Self Cite

View full text Add to dashboard Cite

We describe how to alleviate the problem of plateau phenomena that may arise in learning with a CANFIS neuro-fuzzy modular network. The network model consists of multiple "local-expert" MLPs (multilayer perceptrons) mediated by fuzzy membership functions. Even with such a complex modular architecture, our recently-developed secondorder stagewise backpropagation procedure efficiently evaluates the Hessian matrix of a given objective function, for which we employ the sum-squared-error measure. For concreteness, we use a small curve-fitting problem that allows us to demonstrate detailed analysis based on the Hessian matrix. In particular, we describe how to use a block-diagonal approximate local Hessian matrix associated with a bottleneck local-expert MLP that does not perform very well for a designated task. Since the "bad" performance implies relatively large residuals (or errors), the local Hessian matrix tends to be indefinite; therefore, it is worth exploiting the negative curvature to escape from the plateaus.

show abstract

Section: Discussionsupporting

confidence: 83%

“…(6) in order to exploit sparsity; for more details, see [19], [8], [12], [18]. This is just a block-diagonal approximation to the global Hessian matrix H L .…”

Section: A the Local Hessian Matricesmentioning

confidence: 99%

A block-approximate local Hessian-matrix analysis for CANFIS neuro-fuzzy modular network learning

Mizutani

2012

2012 IEEE International Conference on Fuzzy Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…In recent years, as many important practical problems involve in great number of variables(say, at the magnitude of millions of variables), large scale optimization problems, including large scale nonlinear equations and large scale nonlinear least squares have been attracting more and more attention from researchers, for example see (Toint 1987;Gould et al 2005;Gould and Toint 2007). Most large scale problems have either sparse property or special structure, therefore special approaches, such as partial separability and structure-exploiting, should be and can be applied to such problems (Mizutani and Demmel 2003;Bouaricha and Moré 1997). We consider another approach, using subspace techniques, to large scale problems.…”

Section: Introductionmentioning

confidence: 99%

Subspace methods for large scale nonlinear equations and nonlinear least squares

Yuan

2008

Optim Eng

View full text Add to dashboard Cite

In this paper, we study large scale nonlinear systems of equations and nonlinear least square problems. We present subspace methods for solving these two special optimization problems. The subspace methods have the characteristic to force the next iteration in a low dimensional subspace. The main technique is to construct subproblems in low dimensions so that the computation cost in each iteration can be reduced comparing to standard approaches. The subspace approach offers a possible way to handle large scale optimization problems which are now attracting more and more attention. Actually, quite a few known techniques can be viewed as subspace methods, such as conjugate gradient method, limited memory quasi-Newton method, projected gradient method, and null space method.

show abstract

“…In all of the analyzed cases, the elements causing the most demanding calculations are the positive feedback matrices e i d j , here the complexity is of the order of O(n 2 ). As shown by formulas (15), (20) and in Table 12, the computational complexity O(n) are linked with the other elements of the matrix like c i , c 0 and d i . Interesting results were obtained by juxtaposing formulas describing the computational complexity of the block matrix inversion and the computational complexity resulting from the Gauss method.…”

Section: Discussionmentioning

confidence: 99%

“…64(4) 2016 T. Trawiński, A. Kochan, P. Kielan, and D. Kurzyk Square matrices with entries equal zero except for their main diagonal, one row and one column have many applications e.g. in wireless communication systems [19], neural-network models [20] as well as issues related with chemistry [21] or phisics [22]. In this paper we propose a method of inversion of symetric matrices containing non-zero blocks in their main diagonal, one column, one row and zeros in remaining entries.…”

Section: Introductionmentioning

confidence: 99%

Inversion of selected structures of block matrices of chosen mechatronic systems

Trawiński

Kochan

Kielan

et al. 2016

Bulletin of the Polish Academy of Sciences Technical Sciences

View full text Add to dashboard Cite

Abstract. This paper describes how to calculate the number of algebraic operations necessary to implement block matrix inversion that occurs, among others, in mathematical models of modern positioning systems of mass storage devices. The inversion method of block matrices is presented as well. The presented form of general formulas describing the calculation complexity of inverted form of block matrix were prepared for three different cases of division into internal blocks. The obtained results are compared with a standard Gaussian method and the "inv" method used in Matlab. The proposed method for matrix inversion is much more effective in comparison in standard Matlab matrix inversion "inv" function (almost two times faster) and is much less numerically complex than standard Gauss method.Key words: arrowhead matrices, mechatronic systems, matrix inversion, computational complexity. Abstract. This article describes how to calculate the number of algebraic operations necessary to implement block matrix inversion that occur, among others, in mathematical models of modern positioning systems of mass storage devices. The inversion method of block matrices is presented as well. Presented form of general formulas describing the calculation complexity of inverted form of block matrix were prepared for three different cases of their division into internal blocks. The obtained results are compared with a standard Gaussian method and the "inv" method used in Matlab. The proposed method for matrix inversion is much more effective in comparison in standard Matlab matrix inversion "inv" function (almost two times faster) and is much less numerically complex than standard Gauss method. Inversion of selected structures of block matrices of chosen mechatronic systems

show abstract

On structure-exploiting trust-region regularized nonlinear least squares algorithms for neural-network learning

Cited by 24 publications

References 17 publications

A block-approximate local Hessian-matrix analysis for CANFIS neuro-fuzzy modular network learning

A block-approximate local Hessian-matrix analysis for CANFIS neuro-fuzzy modular network learning

Subspace methods for large scale nonlinear equations and nonlinear least squares

Inversion of selected structures of block matrices of chosen mechatronic systems

Contact Info

Product

Resources

About