Random noise promotes slow heterogeneous synaptic dynamics important for robust working memory computation

Abstract

Recurrent neural networks (RNNs) based on model neurons that communicate via continuous signals have been widely used to study how cortical neurons perform cognitive tasks. Training such networks to perform tasks that require information maintenance over a brief period (i.e., working memory tasks) remains a challenge. Critically, the training process becomes difficult when the synaptic decay time constant is not fixed to a large constant number for all the model neurons. We hypothesize that the brain utilizes intrinsic cortical noise to generate a reservoir of heterogeneous synaptic decay time constants optimal for maintaining information. Here, we show that introducing random, internal noise to the RNNs not only speeds up the training but also produces stable models that can maintain information longer than the RNNs trained without internal noise. Importantly, this robust working memory performance induced by incorporation of internal noise during training is attributed to an increase in synaptic decay time constants of a sub-population of inhibitory units.

Disinhibitory signaling enables flexible coding of top-down information

“…The learning rate was set to 0.01, and the TensorFlow default values were used for the rest of the parameters including the first and second moment decay rates. To further impose biological constraints, we enforced Dale’s law (uniform neurotransmitter release characteristics within separate excitatory and inhibitory neurons) using methods similar to those implemented in previous studies ([43, 49, 74]). To adhere to empirical findings regarding the ratio of excitatory and inhibitory neurons observed in the brain, each RNN consists of 80% excitatory and 20% inhibitory units (i.e., E-I ratio of 80/20; [30, 75, 76]).…”