Gated Linear Networks

Veness, Joel; Lattimore, Tor; Budden, David; Bhoopchand, Avishkar; Mattern, Christopher; Grabska-Barwińska, Agnieszka; Sezener, Eren; Wang, Jianan; Tóth, Peter; Hutter, Marcus

doi:10.1609/aaai.v35i11.17202

Cited by 9 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike contemporary neural networks, the DGN architecture and learning rule is naturally robust to catastrophic forgetting without any modifications or knowledge of task boundaries (something that has been shown for Gated Linear Networks as well [30]). In Fig.…”

Section: Resultsmentioning

confidence: 99%

“…The DGN also differs from and improves over the algorithm on which it was based -the Gated Linear Network (GLN) [29,30]. In particular, the GLN requires a bank of weights for each neuron, with the input choosing which one the neuron should use -something that seems extremely difficult for the brain to implement.…”

Section: Comparison Of Dgns To Other Learning Algorithmsmentioning

confidence: 99%

“…Clipping is used for bounding the loss as well as the gradients, which helps with numerical stability. It also enables a worst-case regret analysis [29,30]. We set to 0.01, so neural activity lies between 0.01 and 0.99.…”

Section: Classification Tasksmentioning

confidence: 99%

“…These include feedback alignment [15,16], creative use of dendrites [17,18], multiplexing [19], and methods in which the error signal is fed directly to each layer rather than propagating backwards from the output layer [20][21][22][23][24][25][26][27][28]. A particularly promising method that falls into the latter category is embodied in Gated Linear Networks [29,30]. These networks, which were motivated from a machine learning rather than a neuroscience perspective, have obtained state-of-the-art results in regression and denoising [31], contextual bandit optimization [32], and transfer learning [33].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A rapid and efficient learning rule for biological neural circuits

Sezener

Grabska-Barwińska

Kostadinov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The dominant view in neuroscience is that changes in synaptic weights underlie learning. It is unclear, however, how the brain is able to determine which synapses should change, and by how much. This uncertainty stands in sharp contrast to deep learning, where changes in weights are explicitly engineered to optimize performance. However, the main tool for doing that, backpropagation, is not biologically plausible, and networks trained with this rule tend to forget old tasks when learning new ones. Here we introduce the Dendritic Gated Network (DGN), a variant of the Gated Linear Network, which offers a biologically plausible alternative to backpropagation. DGNs combine dendritic "gating" (whereby interneurons target dendrites to shape neuronal response) with local learning rules to yield provably efficient performance. They are significantly more data efficient than conventional artificial networks and are highly resistant to forgetting, and we show that they perform well on a variety of tasks, in some cases better than backpropagation. The DGN bears similarities to the cerebellum, where there is evidence for shaping of Purkinje cell responses by interneurons. It also makes several experimental predictions, one of which we validate with in vivo cerebellar imaging of mice performing a motor task.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Comparison Of Dgns To Other Learning Algorithmsmentioning

confidence: 99%

Section: Classification Tasksmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A rapid and efficient learning rule for biological neural circuits

Sezener

Grabska-Barwińska

Kostadinov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This current study extends this work by allowing for plastic corticothalamic connections, and random and fixed thalamocortical connections, adding to the generality of this neural model. Recent modeling work showed a role for multiplicative gating in continual learning problems where it reduces interferences between learned memories and reduces catastrophic forgetting [ 62 , 63 ]. Similar to our method, these models learned to infer task or context boundaries from current inputs, but had a gating mechanism abstracted at a different hierarchical level.…”

Section: Discussionmentioning

confidence: 99%

Thalamic regulation of frontal interactions in human cognitive flexibility

et al. 2022

View full text Add to dashboard Cite

Interactions across frontal cortex are critical for cognition. Animal studies suggest a role for mediodorsal thalamus (MD) in these interactions, but the computations performed and direct relevance to human decision making are unclear. Here, inspired by animal work, we extended a neural model of an executive frontal-MD network and trained it on a human decision-making task for which neuroimaging data were collected. Using a biologically-plausible learning rule, we found that the model MD thalamus compressed its cortical inputs (dorsolateral prefrontal cortex, dlPFC) underlying stimulus-response representations. Through direct feedback to dlPFC, this thalamic operation efficiently partitioned cortical activity patterns and enhanced task switching across different contingencies. To account for interactions with other frontal regions, we expanded the model to compute higher-order strategy signals outside dlPFC, and found that the MD offered a more efficient route for such signals to switch dlPFC activity patterns. Human fMRI data provided evidence that the MD engaged in feedback to dlPFC, and had a role in routing orbitofrontal cortex inputs when subjects switched behavioral strategy. Collectively, our findings contribute to the emerging evidence for thalamic regulation of frontal interactions in the human brain.

show abstract

A unified framework for backpropagation-free soft and hard gated graph neural networks

Pasa,

Navarin,

Erb

et al. 2023

Knowl Inf Syst

View full text Add to dashboard Cite

We propose a framework for the definition of neural models for graphs that do not rely on backpropagation for training, thus making learning more biologically plausible and amenable to parallel implementation. Our proposed framework is inspired by Gated Linear Networks and allows the adoption of multiple graph convolutions. Specifically, each neuron is defined as a set of graph convolution filters (weight vectors) and a gating mechanism that, given a node and its topological context, generates the weight vector to use for processing the node’s attributes. Two different graph processing schemes are studied, i.e., a message-passing aggregation scheme where the gating mechanism is embedded directly into the graph convolution, and a multi-resolution one where neighboring nodes at different topological distances are jointly processed by a single graph convolution layer. We also compare the effectiveness of different alternatives for defining the context function of a node, i.e., based on hyperplanes or on prototypes, and using a soft or hard-gating mechanism. We propose a unified theoretical framework allowing us to theoretically characterize the proposed models’ expressiveness. We experimentally evaluate our backpropagation-free graph convolutional neural models on commonly adopted node classification datasets and show competitive performances compared to the backpropagation-based counterparts.

show abstract

Gated Linear Networks

Cited by 9 publications

References 17 publications

A rapid and efficient learning rule for biological neural circuits

A rapid and efficient learning rule for biological neural circuits

Thalamic regulation of frontal interactions in human cognitive flexibility

A unified framework for backpropagation-free soft and hard gated graph neural networks

Contact Info

Product

Resources

About