Heterogeneous ‘cell types’ can improve performance of deep neural networks

Abstract

Deep convolutional neural networks (CNNs) are powerful computational tools for a large variety of tasks (Goodfellow, 2016). Their architecture, composed of layers of repeated identical neural units, draws inspiration from visual neuroscience. However, biological circuits contain a myriad of additional details and complexity not translated to CNNs, including diverse neural cell types (Tasic, 2018). Many possible roles for neural cell types have been proposed, including: learning, stabilizing excitation and inhibition, and diverse normalization (Marblestone, 2016; Gouwens, 2019). Here we investigate whether neural cell types, instantiated as diverse activation functions in CNNs, can assist in the feed-forward computational abilities of neural circuits. Our heterogeneous cell type networks mix multiple activation functions within each activation layer. We assess the value of mixed activation functions by comparing image classification performance to that of homogeneous control networks with only one activation function per network. We observe that mixing activation functions can improve the image classification abilities of CNNs. Importantly, we find larger improvements when the activation functions are more diverse, and in more constrained networks. Our results suggest a feed-forward computational role for diverse cell types in biological circuits. Additionally, our results open new avenues for the development of more powerful CNNs.

The tuning of tuning: How adaptation influences single cell information transfer

“…The heterogeneity of neuron properties has received much interest lately: for instance, it has been shown that heterogeneity in neural populations can increase coding robustness and efficiency [ 48 ], help optimize information transmission [ 80 ], increase network responsiveness [ 81 ], promote robust learning [ 82 ], help to control the dynamic repertoire of neural populations [ 83 ] and improve the performance on several tasks [ 84 , 85 ]. Here, we show that in particular, the population of excitatory neurons of the barrel cortex shows a large variability in their intrinsic and response properties.…”

The computational and learning benefits of Daleian neural networks

“…Theoretical studies of Deep Neural Networks and their applications have usually ignored Dale's principle, as its inclusion typically impairs learning. Recent results on potential remedies that would allow ANNs to rely on distinct classes of neurons [31] and in particular Excitatory and Inhibitory ones [20] suggest how the integration of design features and properties of real neural networks into artificial ones might be beneficial [32,33]. In particular, the idea that structural constraints on networks can affect learning trajectories and define the loss surface has a long history in constructing and training of artificial neural networks [34][35][36].…”

The tuning of tuning: how adaptation influences single cell information transfer

“…The heterogeneity of neuron properties has received much interest lately: for instance, it has been shown that heterogeneity in neural populations can increase coding robustness and efficiency [47], help optimize information transmission [48], increase network responsiveness [49], promote robust learning [50], help to control the dynamic repertoire of neural populations [51] and improve the performance on several tasks [52, 53]. Here, we show that in particular, the population of excitatory neurons of the barrel cortex shows a large variability in their intrinsic and response properties.…”