On comparison of adaptive regularization methods

Sigurðsson, Sigurður; Larsen, Jan; Hansen, Lars Kai

doi:10.1109/nnsp.2000.889413

Cited by 6 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A large number of research works have been conducted in an attempt to resolve the generalization problem with limited size of training set. Some methods were proposed to improve the generalization ability, including construction (Young and Downs 1998;Kwok and Yeung 1999), pruning (Reed 1993), regularization (Sigurdsson et al 2001), early stopping (Prechelt 1998), noise injection (Wang and Principe 1999), randomly expanded training set (George 2000), and so on.…”

Section: Introductionmentioning

confidence: 99%

The asymptotic optimization of pre-edited ANN classifier

et al. 2009

View full text Add to dashboard Cite

The generalization problem of an artificial neural network (ANN) classifier with unlimited size of training sample, namely asymptotic optimization in probability, is discussed in this paper. As an improved ANN network model, the pre-edited ANN classifier shows better practical performance than the standard one. However, it has not been widely applied due to the absence of the related theoretical support. To further promote its application in practice, the asymptotic optimization of the preedited ANN classifier is studied in this paper. To help study ANN asymptotic optimization in probability, we gives a review of the previous research works on asymptotic optimization in probability of non-parametric classifier, and grouped the main methods into four classes: two-step method, one-step method, generalization method and hypothesis method. In this paper, we adopt generalization/ hypothesis mixed method to prove that pre-edited ANN is asymptotically optimal in probability. Furthermore, a simulation is presented to provide an experimental support for our theoretical work.

show abstract

Section: Introductionmentioning

confidence: 99%

The asymptotic optimization of pre-edited ANN classifier

et al. 2009

View full text Add to dashboard Cite

show abstract

“…A large number of research works have been conducted in an attempt to resolve the generalization problem with limited size of training set. Some methods were proposed to improve the generalization ability, including construction [1][2] , pruning [3] , regularization [4] , early stopping [5] , noise injection [6] , randomly expanded training set [7] , and so on.…”

Section: Introductionmentioning

confidence: 99%

The Asymptotic Optimization of the Pre-edited ANN Classifier

Wang

Yang

Shi

et al. 2008

2008 Fourth International Conference on Natural Computation

View full text Add to dashboard Cite

The generalization problem of an ANN classifier with unlimited size of training sample, namely asymptotic optimization in probability, is discussed in this paper. As an improved ANN network model, the pre-edited ANN classifier shows better practical performance than the standard one. However, no related theoretical research has been conducted on it. To provide a theoretical support for the pre-edited ANN classifier, the asymptotic optimization is studied in this paper. Furthermore, a simulation is presented to provide an experimental support for our theoretical work.

show abstract

“…When applying the regularization method, it is very important to optimize the scale of regularization in order to have a good generalization performance. Various methods have been developed for obtaining the optimal scale of regularization (Sigurdsson, Larsen, & Hansen, 2000). A simple method is to use a validation set for optimizing the scale.…”

Section: Introductionmentioning

confidence: 99%

Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC

2004

View full text Add to dashboard Cite

Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly,the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

show abstract

On comparison of adaptive regularization methods

Cited by 6 publications

References 18 publications

The asymptotic optimization of pre-edited ANN classifier

The asymptotic optimization of pre-edited ANN classifier

The Asymptotic Optimization of the Pre-edited ANN Classifier

Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC

Contact Info

Product

Resources

About