Neural heterogeneity promotes robust learning

Abstract

The brain has a hugely diverse, heterogeneous structure. By contrast, many functional neural models are homogeneous. We compared the performance of spiking neural networks trained to carry out difficult tasks, with varying degrees of heterogeneity. Introducing heterogeneity in membrane and synapse time constants substantially improved task performance, and made learning more stable and robust across multiple training methods, particularly for tasks with a rich temporal structure. In addition, the distribution of time constants in the trained networks closely matches those observed experimentally. We suggest that the heterogeneity observed in the brain may be more than just the byproduct of noisy processes, but rather may serve an active and important role in allowing animals to learn in changing environments.SummaryNeural heterogeneity is metabolically efficient for learning, and optimal parameter distribution matches experimental data.

Heterogeneous ‘cell types’ can improve performance of deep neural networks

“…Beyond dynamical properties, there are many other attributes of neural diversity that are not commonly translated to deep learning, such as cell type specific sensory inputs, excitatory vs inhibitory neurons, and neuromodulation. Our findings join recent studies exploring the computational benefits of neural diversity, such as cell type specific connectivity (Stöckl, 2021), synaptic timescales (Burnham, 2021; Perez-Nieves, 2021), and membrane timescales (Perez-Nieves, 2021). Determining the functional role of neural cell types is an active area of research, and many of these cell type attributes may have translational benefit to deep learning applications.…”

“…Recent studies have classified neurons by their location in the brain, morphology, gene transcription, and electrical properties (Tasic, 2018;Teeter, 2018;Gouwens, 2019;Gouwens, 2020). The biological function of these diverse neural cell types is not yet fully understood, but is an active area of research (Burnham, 2021;Zeldenrust, 2021;Perez-Nieves, 2021). Many possible roles for neural cell types have been proposed, including: learning, stabilizing excitation and inhibition, and diverse normalization (Marblestone, 2016;Gouwens, 2019).…”

“…Many possible roles for neural cell types have been proposed, including: learning, stabilizing excitation and inhibition, and diverse normalization (Marblestone, 2016;Gouwens, 2019). Recently several studies have begun to investigate computational properties of networks with heterogeneous cell types, instantiated in different ways such as excitatory vs inhibitory (Cornford, 2020), synapses (Burnham, 2021), connectivity (Stöckl, 2021), and intrinsic dynamics ( Padmanabhan, 2010;Gjorgjieva, 2016;Duarte, 2019;Perez-Nieves, 2021;Zeldenrust, 2021). Broadly, these studies find computational benefits from adding heterogeneity to neurons.…”

“…Broadly, these studies find computational benefits from adding heterogeneity to neurons. Burnham et al, 2021 andPerez-Nieves et al, 2021 investigated adding heterogeneous synaptic and membrane timescales, finding improved performance on standardized sequential datasets. Stöckl et al, 2021 constructed networks with cell type specific connectivity rules, and found improved performance with reduced number of neurons.…”

“…Here we investigate the effects of adding mixed activation functions as an analog for cell types to an existing CNN architecture. To constrain our investigation we did not explore heterogeneous connectivity (Stöckl, 2021), synapses (Burnham, 2021), timescales (Burnham, 2021 andPerez-Nieves, 2021), or spike-timing (Zeldenrust, 2021). We seek to characterize the network response to the addition of cell types in terms of classification accuracy, learning, and the network's internal representation of the input space.…”