Distributed robust training of multilayer neural netwroks using normalized risk-averting error

Ninomiya, Hiroshi

doi:10.1109/ccmb.2014.7020706

Cited by 3 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the purposes of this work is to model highly nonlinear functions with as few neurons as possible to enable better generalization abilities. It can be observed from Table 1 that 55 is the smallest number of hidden neurons at which a small median and average errors E train (w) < 10 −3 suitable for modeling [18] are obtained. The network structures for the other problems are also determined in similar simulations.…”

Section: Function Approximation Problemsmentioning

confidence: 99%

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

Mahboubi

Indrapriyadarsini

Ninomiya

et al. 2020

Journal of Signal Processing

Self Cite

View full text Add to dashboard Cite

In this paper, we describe a robust technique based on the quasi-Newton method (QN) using an adaptive momentum term to train of neural networks. Microwave circuit models have strong nonlinearities and need a robust training algorithm for their neural network models. The robustness here means that practical solutions can be obtained regardless of the initial values. QN-based algorithms are commonly used for these purposes. Nesterov's accelerated quasi-Newton method (NAQ) proposed a way to accelerate of the QN using a fixed momentum coefficient. In this research, we verify the effectiveness of NAQ for microwave circuit modeling with high nonlinearities and propose a robust QNbased training algorithm with an adaptive momentum coefficient. The proposed algorithm is demonstrated through the modeling of a function and two microwave circuit modeling problems.Here, d p , o p , and w ∈ R n are the pth desired, pth output and weight vectors, respectively. T r denotes the training data set

show abstract

Section: Function Approximation Problemsmentioning

confidence: 99%

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

Mahboubi

Indrapriyadarsini

Ninomiya

et al. 2020

Journal of Signal Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…The problem, called the local minimum problem in training DNNs, has plagued the DNN community since the 1980s 1,2 . DNNs trained with backpropagation are extensively utilized to solve various tasks in artificial intelligence fields for decades [3][4][5][6][7] . The computing power of DNNs is derived through its particularly distributed structure and the capability to learn and generalize.…”

Section: Introductionmentioning

confidence: 99%

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

Wang¹,

Lo²

2020

Preprint

View full text Add to dashboard Cite

This paper aims to overcome a fundamental problem in the theory and application of deep neural networks (DNNs). We propose a method to solve the local minimum problem in training DNNs directly. Our method is based on the cross-entropy loss criterion's convexification by transforming the cross-entropy loss into a risk averting error (RAE) criterion. To alleviate numerical difficulties, a normalized RAE (NRAE) is employed. The convexity region of the cross-entropy loss expands as its risk sensitivity index (RSI) increases. Making the best use of the convexity region, our method starts training with an extensive RSI, gradually reduces it, and switches to the RAE as soon as the RAE is numerically feasible. After training converges, the resultant deep learning machine is expected to be inside the attraction basin of a global minimum of the cross-entropy loss. Numerical results are provided to show the effectiveness of the proposed method.

show abstract

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

Wang¹,

Lo²

2021

View full text Add to dashboard Cite

Distributed robust training of multilayer neural netwroks using normalized risk-averting error

Cited by 3 publications

References 17 publications

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

Contact Info

Product

Resources

About