A Unifying View of Gradient Calculations and Learning for Locally Recurrent Neural Networks

Abstract

In this paper a critical review of gradient-based training methods for recurrent neural networks is presented including Back Propagation Through Time (BPTT), Real Time Recurrent Learning (RTRL) and several specific learning algorithms for different locally recurrent architectures. From this survey it comes out the need for a unifying view of all the specific procedures proposed for networks with local feedbacks, that keeps into account the general framework of recurrent networks learning: BPTT and RTRL. Therefore a learning method for local feedback network is proposed which combines together the best feature of BPTT, i.e. the lowest complexity, and of RTRL, i.e. the on-line operation, and includes as special case several specific algorithms already proposed, such as Temporal Back Propagation, Back Propagation for Sequences, Back-Tsoi algorithm and some others. In the general version, this new training method allows on-line efficient and accurate gradient calculation. It compares favourably with the previous algorithms in stability, speed/complexity trade off, accuracy.

New second-order algorithms for recurrent neural networks based on conjugate gradient

“…After using these algorithms with some static problems, with good performances, we have applied them to training locally recurrent neural networks, as explained in the next section. [6,7], and which has good performances.…”

“…With the same notation used in [6,7,8] As stated above, applying the tW0 previous techniques to calculate the products H(%)p , we obtain two new algorithms based on the conjugate gradient method.…”

Dynamical systems learning by a circuit theoretic approach

“…The obtained gradient information can be used both for parameter sensitivity computation of systems and for training of adaptive systems. The system can be any causal, in general non-linear and time-variant, dynamical system represented by a SFG, in particular any feedforward (static), time delay or recurrent neural networks [2,3,9]. In this work we use discrete time notation, but the same theory holds for the continuous time case.…”

“…On-line training of adaptive systems is very important in real applications such as Digital Signal Processing, system identification and control, channel equalization and predistortion, since it allows the model to follow time-varying features of the real system [2,3]. By on-line learning in contrast to batch learning this is provided with no interruption of the operation of the system (such as a communication channel).…”

“…In this work we use discrete time notation, but the same theory holds for the continuous time case. The gradient is obtained by the analysis of two SFGs, the original one and its adjoint, without the complex chain rule expansions of derivatives usually employed [2,3,9]. This method can be used also for off-line learning.…”