An Approach to Stable Gradient-Descent Adaptation of Higher Order Neural Units

Abstract

Abstract-Stability evaluation of a weight-update system of higher-order neural units (HONUs) with polynomial aggregation of neural inputs (also known as classes of polynomial neural networks) for adaptation of both feedforward and recurrent HONUs by a gradient descent method is introduced. An essential core of the approach is based on spectral radius of a weight-update system, and it allows stability monitoring and its maintenance at every adaptation step individually. Assuring stability of the weight-update system (at every single adaptation step) naturally results in adaptation stability of the whole neural architecture that adapts to target data. As an aside, the used approach highlights the fact that the weight optimization of HONU is a linear problem, so the proposed approach can be generally extended to any neural architecture that is linear in its adaptable parameters.

Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures With Gradient Learnings

“…Thus, the previous sections thoroughly prove that the sufficient condition to maintain the learning convergence under ISS's umbrella and in the sense of BIBS stability is (20) that yields (21) for non-momentum gradient algorithms. For experimental results, e.g., with a class of polynomial neural architectures, please see paper [10], where spectral radii's effect ( ) A  on gradient learning is studied and shown in detail. Also, this letter's proofs are the theoretical complements to the earlier, i.e., more experimentally focused work [10] that did not explicitly show the whole theoretical kinship.…”

“…For experimental results, e.g., with a class of polynomial neural architectures, please see paper [10], where spectral radii's effect ( ) A  on gradient learning is studied and shown in detail. Also, this letter's proofs are the theoretical complements to the earlier, i.e., more experimentally focused work [10] that did not explicitly show the whole theoretical kinship. The BIBS condition (20) can be restated as…”

“…that is very strict, and also, the norm must not necessarily be kept all the time below 1 as it practically fluctuates around 1 as also shown in [10]. Thus, it is more practical to maintain the following condition…”

“…By IPLNAs in this paper, we consider a wide family of shallow neural networks including the extreme learning machine (ELM) or random vector functional link (RVFL) networks [1]- [3], other functional links and kernel neural architectures and filters, e.g., [4]- [6], and polynomial neural networks [7], [8] including basic standalone  structures called polynomial neural units in [9], [10], and references therein. This letter shows that the input-to-state stability (ISS) concept [11] and BIBS stability [12] generally apply to the gradient learning algorithms and their many modifications for IPLNA's.…”

Slip and Adhesion in a Railway Wheelset Simulink Model Proposed for Detection Driving Conditions Via Neural Networks

“…The gradient based optimization algorithms, as the Gradient Descent, are usually used for the HONU training. [11] After the first part, where the data for driving conditions evaluation are prepared, the second part comes, where the in first part prepared vector of HONU weights is fed to suitable clustering algorithm for clusters training or evaluation. The most widespread clustering algorithms are based on the K-means algorithm [12].…”