Training stochastic stabilized supralinear networks by dynamics-neutral growth

Abstract

There continues to be a trade-off between the biological realism and performance of neural networks. Contemporary deep learning techniques allow neural networks to be trained to perform challenging computations at (near) human-level, but these networks typically violate key biological constraints. More detailed models of biological neural networks can incorporate many of these constraints but typically suffer from subpar performance and trainability. Here, we narrow this gap by developing an effective method for training a canonical model of cortical neural circuits, the stabilized supralinear network (SSN), that in previous work had to be constructed manually or trained with undue constraints. SSNs are particularly challenging to train for the same reasons that make them biologically realistic: they are characterized by strongly-connected excitatory cells and expansive firing rate non-linearities that together make them prone to dynamical instabilities unless stabilized by appropriately tuned recurrent inhibition. Our method avoids such instabilities by initializing a small network and gradually increasing network size via the dynamics-neutral addition of neurons during training. We first show how SSNs can be trained to perform typical machine learning tasks by training an SSN on MNIST classification. We then demonstrate the effectiveness of our method by training an SSN on the challenging task of performing amortized Markov chain Monte Carlo-based inference under a Gaussian scale mixture generative model of natural image patches with a rich and diverse set of basis functions – something that was not possible with previous methods. These results open the way to training realistic cortical-like neural networks on challenging tasks at scale.

Synapse-type-specific competitive Hebbian learning forms functional recurrent networks

“…= J * (w − w * ), [21] where w * is zero, by definition, and J * is the Jacobian evaluated at the fixed point. The Jacobian is defined as…”

“…Theoretical work has highlighted the experimentally observed balance of stimulus selective excitatory and inhibitory input currents as a critical requirement for many neural computations (11)(12)(13)(14)(15)(16). For example, recent models based on balanced E-I networks can explain a wide range of cortical phenomena, such as cross-orientation and surround suppression (17,18), as well as stimulus-induced neural variability (19)(20)(21). A major caveat of these models is that the network connectivity is usually static and designed by hand, albeit based on experimental measurements.…”

Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies

“…Other common choices include softmax, softplus or linear, depending on the decoding objective. These models reflect the diversity in function across the brain areas they model, such perceptual functions in early sensory areas such as the visual cortex [9], maintenance of items in working memory, decision-making, context-dependent or rule-dependent responses in higher-order cortical areas such as the prefrontal cortex [14,16,19], and motor control of eye saccades or arm movements in motor cortical areas [10][11][12][13]. Consequently, the task structure, requisite computations, RNN inputs and outputs vary widely according to the function.…”

“…First, the brain's connectivity is highly recurrent at multiple spatial scales, making recurrent neural networks (RNNs) [5] the modeling tool of choice in neuroscience [2,4,6]. The training and analysis of RNNs has advanced theory on the neural basis of perception [7][8][9], motor behavior [10][11][12][13], cognition [14][15][16][17][18][19][20][21][22][23][24][25][26][27], memory and navigation [28,29]. Second, physiological properties of neurons places a greater demand on the emergent computational properties of neural populations.…”

“…Consequently, the task structure, requisite computations, RNN inputs and outputs vary widely according to the function. We review the values of some of these parameters from previously-published models [9, 11–25, 29, 48–60] in Appendix C and Table S1 in the supplementary.…”