Oriented stochastic loss descent algorithm to train very deep multi-layer neural networks without vanishing gradients

Abuqaddom, Inas; Mahafzah, Basel A.; Faris, Hossam

doi:10.1016/j.knosys.2021.107391

Cited by 52 publications

(23 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other techniques have incorporated federated learning and momentum [81] and used evolutionary algorithms [82], speculative approaches [83] and spiking neural network concepts [84,85]. Yet other techniques have focused on supporting deep networks [86], memory use optimization [87], bias factors [88,89] and initial condition sensitivity [90]. A recent technique, proposed by Zhang, et al [91], utilizes a combination of expert strategies and gradient descent for optimization.…”

Section: Gradient Descentmentioning

confidence: 99%

Automating the design and development of gradient descent trained expert system networks

Straub¹

2022

Knowledge-Based Systems

View full text Add to dashboard Cite

Section: Gradient Descentmentioning

confidence: 99%

Automating the design and development of gradient descent trained expert system networks

Straub¹

2022

Knowledge-Based Systems

View full text Add to dashboard Cite

“…To overcome the vanishing-gradient problem, Inas et al. [ 28 ] proposed OSLD, a new anti-vanishing back-propagated learning algorithm. Prior works [ 29 ] directly train SNNs using backpropagation; however, it is insufficient when training spiking architectures of the size of VGG-16 [ 7 ].…”

Section: Background and Related Workmentioning

confidence: 99%

Optimal Mapping of Spiking Neural Network to Neuromorphic Hardware for Edge-AI

Xiao

Chen

Wang

2022

Sensors

View full text Add to dashboard Cite

Neuromorphic hardware, the new generation of non-von Neumann computing system, implements spiking neurons and synapses to spiking neural network (SNN)-based applications. The energy-efficient property makes the neuromorphic hardware suitable for power-constrained environments where sensors and edge nodes of the internet of things (IoT) work. The mapping of SNNs onto neuromorphic hardware is challenging because a non-optimized mapping may result in a high network-on-chip (NoC) latency and energy consumption. In this paper, we propose NeuMap, a simple and fast toolchain, to map SNNs onto the multicore neuromorphic hardware. NeuMap first obtains the communication patterns of an SNN by calculation that simplifies the mapping process. Then, NeuMap exploits localized connections, divides the adjacent layers into a sub-network, and partitions each sub-network into multiple clusters while meeting the hardware resource constraints. Finally, we employ a meta-heuristics algorithm to search for the best cluster-to-core mapping scheme in the reduced searching space. We conduct experiments using six realistic SNN-based applications to evaluate NeuMap and two prior works (SpiNeMap and SNEAP). The experimental results show that, compared to SpiNeMap and SNEAP, NeuMap reduces the average energy consumption by 84% and 17% and has 55% and 12% lower spike latency, respectively.

show abstract

“…A vanishing gradient causes the weight tuning to be extremely small, such that the network inefficiently corrects the computation errors for a close mapping between the inputs and responses. [37] f…”

Section: Artificial Neural Network (Ann)mentioning

confidence: 99%

f (x) = \frac{((), e^{x} goodbreak- e^{- x})}{((), e^{x} goodbreak+ e^{- x})}

where

e

is the base of the natural logarithm constant and

x

is the input attribute. [ 36 ]…”

Section: Introductionmentioning

confidence: 99%

The use of artificial neural network (ANN) in dry flue gas desulphurization modelling: Levenberg–Marquardt (LM) and Bayesian regularization (BR) algorithm comparison

et al. 2022

View full text Add to dashboard Cite

This research project aims to investigate the efficacy of artificial neural networks (ANN) in mapping dry flue gas desulphurization (DFGD). Bayesian regularization (BR) and Levenberg-Marquardt (LM) training algorithms were used for DFGD modelling. The input layer feed data contained diatomite to Ca (OH) 2 ratio, hydration time, hydration temperature, sulphation temperature, and inlet gas concentration, while the output layer metadata were sorbent conversion and sulphation responses. The hyperbolic tangent (tansig), sigmoid (logsig), and linear (purelin) activation functions were compared to ascertain the best network learning model. The number of hidden layer cells also varied between 7 and 10, given the existence of multiple output feed data. BR and LM performance evaluation was based on coefficient of determination (R 2 ), root mean square error (RMSE), and mean square error (MSE) mathematical analysis. BR was a superlative training tool compared to LM, with lower RMSE and MSE values. The goodness of fit data for both techniques was close to unity, clarifying that ANN using BR and LM tools can be used to predict DGFD outcome. The shrinking core model was used to analyze the desulphurization reaction and concluding the chemical reaction was the reaction controlling step.

show abstract

Oriented stochastic loss descent algorithm to train very deep multi-layer neural networks without vanishing gradients

Cited by 52 publications

References 14 publications

Automating the design and development of gradient descent trained expert system networks

Automating the design and development of gradient descent trained expert system networks

Optimal Mapping of Spiking Neural Network to Neuromorphic Hardware for Edge-AI

The use of artificial neural network (ANN) in dry flue gas desulphurization modelling: Levenberg–Marquardt (LM) and Bayesian regularization (BR) algorithm comparison

Contact Info

Product

Resources

About