Adaptive time scales in recurrent neural networks

Abstract

Recurrent neural network models have become widely used in computational neuroscience to model the dynamics of neural populations as well as in machine learning applications to model data with temporal dependencies. The different variants of RNNs commonly used in these scientific fields can be derived as discrete time approximations of the instantaneous firing rate of a population of neurons. The time constants of the neuronal process are generally ignored in these approximations, while learning these time constants could possibly inform us about the time scales underlying temporal processes and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard Elman approximation to a more lenient one that still accounts for the time scales at which processes unfold. We show that such a model with adaptive time scales performs better on predicting temporal data, increasing the memory capacity of recurrent neural networks, and allows recovery of the time scales at which the underlying processes unfold.

Constructing and Forgetting Temporal Context in the Human Cerebral Cortex

“…Our computational approach was inspired by the neurocognitive models of Botvinick, (2007) and Kiebel et al (2008), in which higher stages of cortical processing learned or controlled temporal structure at longer timescales. More generally, multiscale machine-learning architectures have been proposed for reducing the complexity of the learning problem at each scale and for representing multi-scale environments (Chung et al, 2016;Jaderberg et al, 2019;Mozer, 1992;Mujika et al, 2017;Quax et al, 2019;Schmidhuber, 1992). In neuroscience, multiple timescale representations have been proposed for learning value functions (Sutton, 1995), tracking reward (Bernacchia et al, 2011), and perceiving and controlling action (Botvinick, 2007;Paine and Tani, 2005).…”

Neural heterogeneity promotes robust learning

“…A number of studies have used heterogeneous or tunable time constants (Fang et al, 2020;Quax et al, 2020;Yin et al, 2020), but these have generally been focussed on maximising performance for neuromorphic applications, and not considering the potential role in real nervous systems. In particular, we have shown that: heterogeneity is particularly important for the type of temporally complex tasks faced in real environments, as compared to the static ones often considered in machine learning; heterogeneity confers robustness allowing for learning in a wide range of environments; optimal distributions of time constants are consistent across training runs and match experimental data; and that our results are not specific to a particular task or training method.…”

A stacked autoencoder‐based convolutional and recurrent deep neural network for detecting cyberattacks in interconnected power control systems

“…As it will be shown below, the usage of several stacked convolutional layers in CNNs, used to create a hierarchy of progressively more abstract representations of the input data, constitutes their main strength for mining spatial dependencies. Similarly, recurrent neural networks (RNNs) are the most suitable neural network typology for modeling temporal dependencies in time‐series data 21 …”