Adaptive Regularization in Neural Network Modeling

Larsen, Jan; Svarer, Claus; Andersen, Lars Nonboe; Hansen, Lars Kai

doi:10.1007/978-3-642-35289-8_8

Cited by 25 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, Ref. [34][35][36][37][38][39] proposed many adaptive techniques for automatically updating continuous hyperparameters, such as the learning rate momentum and weight decay for each iteration to improve the coverage speed of backpropagation. In addition, early stopping [40,41] can be used when the error rate on a validation set or training set has not improved, or when the error rate increases for a number of epochs.…”

Section: Related Workmentioning

confidence: 99%

A Framework for Designing the Architectures of Deep Convolutional Neural Networks

Albelwi

Mahmood

2017

Entropy

210

View full text Add to dashboard Cite

Recent advances in Convolutional Neural Networks (CNNs) have obtained promising results in difficult deep learning tasks. However, the success of a CNN depends on finding an architecture to fit a given problem. A hand-crafted architecture is a challenging, time-consuming process that requires expert knowledge and effort, due to a large number of architectural design choices. In this article, we present an efficient framework that automatically designs a high-performing CNN architecture for a given problem. In this framework, we introduce a new optimization objective function that combines the error rate and the information learnt by a set of feature maps using deconvolutional networks (deconvnet). The new objective function allows the hyperparameters of the CNN architecture to be optimized in a way that enhances the performance by guiding the CNN through better visualization of learnt features via deconvnet. The actual optimization of the objective function is carried out via the Nelder-Mead Method (NMM). Further, our new objective function results in much faster convergence towards a better architecture. The proposed framework has the ability to explore a CNN architecture's numerous design choices in an efficient way and also allows effective, distributed execution and synchronization via web services. Empirically, we demonstrate that the CNN architecture designed with our approach outperforms several existing approaches in terms of its error rate. Our results are also competitive with state-of-the-art results on the MNIST dataset and perform reasonably against the state-of-the-art results on CIFAR-10 and CIFAR-100 datasets. Our approach has a significant role in increasing the depth, reducing the size of strides, and constraining some convolutional layers not followed by pooling layers in order to find a CNN architecture that produces a high recognition performance.

show abstract

Section: Related Workmentioning

confidence: 99%

A Framework for Designing the Architectures of Deep Convolutional Neural Networks

Albelwi

Mahmood

2017

Entropy

210

View full text Add to dashboard Cite

show abstract

“…the K-fold cross-validation [19], leads to an adaptive regularization scheme originally suggested in [12], which was further improved in [l], [3], [5], [13]. Suppose that all available data 2) = { z ( k ) ; y(k)}F=, of N input-output examples, split into K randomly chosen disjoint sets of approximately equal size, i.e., 2) = U g l V j and V i # j : V; n V j = 8. a a i n i n g and validation is replicated K times, and in the j'th run training is done on the set 7; = V\Vj and validation is performed on V j .…”

Section: Validation Error Approachmentioning

confidence: 99%

“…N,j is number of validation examples. Fcv is an estimate of the average generalization error over all possible training sets of size N t j , see [13].…”

Section: Validation Error Approachmentioning

confidence: 99%

“…In case of linear regularization, the gradient of the cross-validation error can be written as [12], [13] aEv 1 asv.…”

Section: Validation Error Approachmentioning

confidence: 99%

“…Some methods focus directly on minimizing an estimate of the generalization error (either algebraic or empirical) [l], [3], [5], [6], [7], [12], [13], whereas others starts from different criteria, e.g., the Bayesian evidence [2, Ch. lo], [7], [15], [16].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On comparison of adaptive regularization methods

Sigurðsson

Larsen²,

Hansen³

Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501)

View full text Add to dashboard Cite

Adaptive Sampling for Surrogate Modelling with Artificial Neural Network and its Application in an Industrial Cracking Furnace

Jin

et al. 2016

Can J Chem Eng

View full text Add to dashboard Cite

In surrogate modelling, a simple functional approximation of a complex system model is always constructed to reduce the computational expense, and the selection of a suitable surrogate model and a sampling method are key to obtaining a surrogate model for a complex system. To construct an appropriate surrogate model, three methods of adaptive surrogate modelling that use artificial neural networks (ANN) are developed by incorporating a new mechanism for automatically determining the number of hidden nodes and/or a new prediction error-based mixed adaptive sampling method. In the automatic determination, the number of hidden nodes can adaptively change according to the effective rate of parameters in the ANN during the adaptive surrogate modelling process. As a result, an improper number of hidden nodes determined by the empirical method can be avoided. The prediction error-based mixed adaptive sampling method is capable of finding the strong nonlinear behaviour of the underlying system, which is easily missed by the traditional prediction variance-based sampling method. The three methods and the previous method for adaptive surrogate modelling that use ANN are tested and compared in terms of replicating the behaviours of three types of challenge functions to determine the efficacy of the developed methods. Furthermore, these methods are used in an engineering problem of surrogate modelling for a cracking reaction simulator to validate the efficacy of the developed methods.

show abstract

Adaptive Regularization in Neural Network Modeling

Cited by 25 publications

References 28 publications

A Framework for Designing the Architectures of Deep Convolutional Neural Networks

A Framework for Designing the Architectures of Deep Convolutional Neural Networks

On comparison of adaptive regularization methods

Adaptive Sampling for Surrogate Modelling with Artificial Neural Network and its Application in an Industrial Cracking Furnace

Contact Info

Product

Resources

About