Markovian Architectural Bias of Recurrent Neural Networks

Abstract

In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training [1], [2]. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure. Index Terms-Complex symbolic sequences, information latching problem, iterative function systems, Markov models, recurrent neural networks (RNNs).

State Estimation for Discrete-Time Neural Networks with Markov-Mode-Dependent Lower and Upper Bounds on the Distributed Delays

“…In the context of Markovian jumping RNNs, the exponential stability has been studied in many papers, see e.g. [1,3,15,17,21,25,35]. In particular, the asymptotical stability has been investigated in [25] for continuous-time RNNs with Markovian jumping parameters, and stability analysis and synchronization problems have been dealt with in [15] for a class of discrete-time Markovian jumping RNNs with mixed time-delays.…”

Comparison of Echo State Networks with Simple Recurrent Networks and Variable-Length Markov Models on Symbolic Sequences

“…As pointed out in [10], the state space of RNNs initialized with small weights is organized in Markovian way prior to any training. To assess, what has been actually learnt during the training process it is always necessary to compare performance of the trained RNNs with Markov models.…”

“…Several connectionist models directly using Markovian organization [10] of the RNN's state space were suggested. Activities of recurrent neurons in an recurrent neural network initialized with small weights are grouped in clusters [9].…”

“…The second dataset contains words generated by simple context free grammar. The structure and the recursion depths are fully controlled by the designer [10] in this case.…”

“…Some researchers realized that even in an untrained randomly initialized recurrent network considerable amount of clustering is present. This was first explained in [9] and correspondence to a class of variable length Markov models was shown in [10].…”