Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures With Gradient Learnings

Abstract

This letter summarizes and proves the concept of bounded-input bounded-state (BIBS) stability for weight convergence of a broad family of in-parameter-linear nonlinear neural architectures as it generally applies to a broad family of incremental gradient learning algorithms. A practical BIBS convergence condition results from the derived proofs for every individual learning point or batches for real-time applications.

Living Lab Long-Term Sustainability in Hybrid Access Positive Energy Districts—A Prosumager Smart Fog Computing Perspective

“…On the optimal information processing and distribution side, we utilize a class of shallow neural architectures that can be used as suitable learning predictors in the sense of fog computing [36]. These neural architectures are polynomial neural architectures, that is, Higher-Order Neural Units (HONU) [37], [38] with customized order of polynomial are a subclass of In-Parameter-Linear-Nonlinear Architectures (IPLNAs) [38] that have intriguing properties regarding computational efficiency and simultaneous weight convergence assurance [39] and stability monitoring of the underlying system represented by the training data. Recall that HONUs can also be viewed as energy-saving computing machine learning tools due to their properties and abilities to run efficiently on low power consumption devices.…”

“…where µ is the learning rate that can vary over time depending on the applied form of learning, and Q is the error criterion Q(k) = e(k) 2 . It is shown in [39] that for IPLNAs, including HONUs, the weight-update system in (7) can be expressed in the linear time-variant state-space representation form.…”

“…determine the BIBS stability of the weight system (7) during learning (see [39] for further details). The relevant experimental study can be found in [38], where the most strict condition of the spectral radius ρ (A(k)) < 1 is demonstrated when weight convergence is evaluated in every individual data sample (of a time series).…”

“…The relevant experimental study can be found in [38], where the most strict condition of the spectral radius ρ (A(k)) < 1 is demonstrated when weight convergence is evaluated in every individual data sample (of a time series). Newly, we highlight the connotation of the weight convergence rule [39] with the famous Levenberg-Marquardt batch learning algorithm (LM, with origins in [53], [54]) as follows.…”

“…For feedforward IPLNAs, the weight-update system (13) is a linear timeinvariant dynamical system (LTI) because the Jacobian J is constant with respect to the epoch index ϵ. For recurrent IPLNAs with the L-M algorithm, system (13) is a dynamical linear time variable (LTV) system, and the BIBS stability paradigm shall be used as in [39].…”