Continual Learning with Deep Artificial Neurons

Abstract

Neurons in real brains are enormously complex computational units. Among other things, they're responsible for transforming inbound electro-chemical vectors into outbound action potentials, updating the strengths of intermediate synapses, regulating their own internal states, and modulating the behavior of other nearby neurons. One could argue that these cells are the only things exhibiting any semblance of real intelligence. It is odd, therefore, that the machine learning community has, for so long, relied upon the assumption that this complexity can be reduced to a simple sum and fire operation. We ask, might there be some benefit to substantially increasing the computational power of individual neurons in artificial systems? To answer this question, we introduce Deep Artificial Neurons (DANs), which are themselves realized as deep neural networks. Conceptually, we embed DANs inside each node of a traditional neural network, and we connect these neurons at multiple synaptic sites, thereby vectorizing the connections between pairs of cells. We demonstrate that it is possible to meta-learn a single parameter vector, which we dub a neuronal phenotype, shared by all DANs in the network, which facilitates a meta-objective during deployment. Here, we isolate continual learning as our meta-objective, and we show that a suitable neuronal phenotype can endow a single network with an innate ability to update its synapses with minimal forgetting, using standard backpropagation, without experience replay, nor separate wake/sleep phases. We demonstrate this ability on sequential non-linear regression tasks. * first author Preprint. Under review.

Optimizing Reusable Knowledge for Continual Learning via Metalearning

“…Works similar to ours have proposed to find a path of relevant weights to solve the task [38], freezing used weights, and limiting learning of new tasks. Others use different functions as components in the network, either Hypernetworks [39], Deep Artificial Neurons (DANs) [40], or Compositional Structures [41], so that the network components are more flexible during learning.…”

Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks

“…They showed that a single neuron modeled as a dendritic ANN with sparse connectivity could solve binary classification tasks (e.g., MNIST) and benefit from having the same input at different dendritic sites. Another approach utilizes multiple dendritic layers in each node of a deep ANN, and reduces catastrophic forgetting in a continual learning task [70]. To explore the potential advantages of human dCaAPs, a recent study used their respective transfer function in various deep learning architectures and found a small but significant increase in classification performance for various tasks, especially when used in combination with sparse connectivity resembling that of dendritic trees [71].…”