Recent experiments have revealed a hierarchy of time scales in the visual cortex, where different stages of the visual system process information at different time scales. Recurrent neural networks are ideal models to gain insight in how information is processed by such a hierarchy of time scales and have become widely used to model temporal dynamics both in machine learning and computational neuroscience. However, in the derivation of such models as discrete time approximations of the firing rate of a population of neurons, the time constants of the neuronal process are generally ignored. Learning these time constants could inform us about the time scales underlying temporal processes in the brain and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard approximations to a more lenient one that accounts for the time scales at which processes unfold. We show that such a model performs better on predicting simulated neural data and allows recovery of the time scales at which the underlying processes unfold. A hierarchy of time scales emerges when adapting to data with multiple underlying time scales, underscoring the importance of such a hierarchy in processing complex temporal information.

Recurrent neural network models have become widely used in computational neuroscience to model the dynamics of neural populations as well as in machine learning applications to model data with temporal dependencies. The different variants of RNNs commonly used in these scientific fields can be derived as discrete time approximations of the instantaneous firing rate of a population of neurons. The time constants of the neuronal process are generally ignored in these approximations, while learning these time constants could possibly inform us about the time scales underlying temporal processes and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard Elman approximation to a more lenient one that still accounts for the time scales at which processes unfold. We show that such a model with adaptive time scales performs better on predicting temporal data, increasing the memory capacity of recurrent neural networks, and allows recovery of the time scales at which the underlying processes unfold.