Learning long-term dependencies in NARX recurrent neural networks

Abstract

It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.

Application of neural networks with orthogonal activation functions in control of dynamical systems

“…However, learning algorithms for recurrent neural networks can perform poorly in capturing the global behaviour of the system. Lin, Horne, Tino, and Giles (1996) proposed the NARX model to solve this problem. In Godarzi, Amiri, Talaei, and Jamasb (2014), NARX is presented as an advanced type of recurrent neural network, which considers the factor of time (which is of great importance, because MLS is a time-varying system).…”

Nonlinear modeling and control approach to magnetic levitation ball system using functional weight RBF network-based state-dependent ARX model

“…An important class of discrete-time nonlinear model is the nonlinear AutoRegressive model with eXogenous variables (NARX) model [23] as follows yðtÞ ¼ f ðyðt À 1Þ; yðt À 2Þ; …; yðt À n y Þ; uðt À 1Þ; uðt À 2Þ; …; uðt À n u ÞÞ þ ξðtÞ ð 1Þ…”

Comparison of artificial neural networks and harmonic analysis for sea level forecasting (Urias coastal lagoon, Mazatlan, Mexico)

“…Para el segundo método usamos modelos autorregresivos no lineales (NAR, por sus siglas en inglés) basados en redes neuronales artificiales (ANN, por sus siglas en inglés) (Lin et al 1996, Haykin 1999. En este enfoque se define un modelo dinámico que requiere un conjunto de condiciones iniciales, que son valores pasados de la serie de tiempo y se usan para predecir los valores futuros.…”

“…For the second method, we used nonlinear autoregressive (NAR) models based on artificial neural networks (ANN) (Lin et al 1996, Haykin 1999. In this approach, one defines a dynamical model that requires a set of initial conditions, which are past values of the time series, used to predict future values.…”