Dendritic normalisation improves learning in sparsely connected artificial neural networks

Abstract

Artificial neural networks, taking inspiration from biological neurons, have become an invaluable tool for machine learning applications. Recent studies have developed techniques to effectively tune the connectivity of sparsely-connected artificial neural networks, which have the potential to be more computationally efficient than their fully-connected counterparts and more closely resemble the architectures of biological systems. We here present a normalisation, based on the biophysical behaviour of neuronal dendrites receiving distributed synaptic inputs, that divides the weight of an artificial neuron’s afferent contacts by their number. We apply this dendritic normalisation to various sparsely-connected feedforward network architectures, as well as simple recurrent and self-organised networks with spatially extended units. The learning performance is significantly increased, providing an improvement over other widely-used normalisations in sparse networks. The results are two-fold, being both a practical advance in machine learning and an insight into how the structure of neuronal dendritic arbours may contribute to computation.

A GPU-based computational framework that bridges neuron simulation and artificial intelligence

“…Jones and Kording 91 used a binary tree to approximate dendrite branching and provided valuable insights into the influence of tree structure on single neurons’ computational capacity. Bird et al 92 . proposed a dendritic normalization rule based on biophysical behavior, offering an interesting perspective on the contribution of dendritic arbor structure to computation.…”

Invariant synaptic density across species

“…The output spiking rate R can then be approximated by R ≈ IF syn + c, where IF syn represents the input fraction of activated synapses (as a percentage of total postsynapses) and c is a constant (27). Such connectivity and morphology independent point-like neurons endow networks with the capacity to maintain functional robustness due to normalisation effects (28)(29)(30), paralleling methods employed in machine learning (31,32).…”

Distinctive properties of biological neural networks and recent advances in bottom-up approaches toward a better biologically plausible neural network

“…Because of the additional nonlinearity compared to a model point neuron, better expressibility can be expected (Wu et al, 2018 ), and electrical compartmentalization and active dendritic properties can be applied to ANNs (Chavlis and Poirazi, 2021 ; Iyer et al, 2022 ; Sezener et al, 2022 ). The segregated electrical properties also indicate that homeostatic control can occur separately in distinct dendritic branches (Tripodi et al, 2008 ; Bird et al, 2021 ; Shen et al, 2021 ). Such an adjustment of weights in each dendritic branch toward a certain homeostatic level is similar to the normalization step in ANN (Shen et al, 2021 ), which also improves learning in sparsely connected neural networks, such as BNN (Bird et al, 2021 ).…”

“…The segregated electrical properties also indicate that homeostatic control can occur separately in distinct dendritic branches (Tripodi et al, 2008 ; Bird et al, 2021 ; Shen et al, 2021 ). Such an adjustment of weights in each dendritic branch toward a certain homeostatic level is similar to the normalization step in ANN (Shen et al, 2021 ), which also improves learning in sparsely connected neural networks, such as BNN (Bird et al, 2021 ). The typical structure of a cortical pyramidal neuron consists of two distinctive directions of dendritic outgrowth from the soma: basal and apical dendrites (DeFelipe and Farias, 1992 ).…”