Neuroscience-Inspired Online Unsupervised Learning Algorithms: Artificial Neural Networks

Pehlevan, Cengiz; Chklovskii, Dmitri B.

doi:10.1109/msp.2019.2933846

Cited by 43 publications

(29 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While not explicitly our goal here, translating principles of neural computation to improve machine learning is a long-sought goal at the interface of computer science and neuroscience (39,(95)(96)(97). Specifically, these insights may be applicable for deep learning (98) [e.g., to enhance attentionmodulated artificial neural networks (99,100)], for online similarity search problems, and in electronic noses for event-based smell processing.…”

Section: Discussionmentioning

confidence: 99%

Habituation as a neural algorithm for online odor discrimination

Shen

Dasgupta

Navlakha

2020

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Habituation is a form of simple memory that suppresses neural activity in response to repeated, neutral stimuli. This process is critical in helping organisms guide attention toward the most salient and novel features in the environment. Here, we follow known circuit mechanisms in the fruit fly olfactory system to derive a simple algorithm for habituation. We show, both empirically and analytically, that this algorithm is able to filter out redundant information, enhance discrimination between odors that share a similar background, and improve detection of novel components in odor mixtures. Overall, we propose an algorithmic perspective on the biological mechanism of habituation and use this perspective to understand how sensory physiology can affect odor perception. Our framework may also help toward understanding the effects of habituation in other more sophisticated neural systems.

show abstract

Section: Discussionmentioning

confidence: 99%

Habituation as a neural algorithm for online odor discrimination

Shen

Dasgupta

Navlakha

2020

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…More Neuroscience-oriented approaches attempt to find which learning rules could implement a biologically plausible version of backpropagation [26,27]. In contrast to most works described previously relying on numerical optimization to find learning rules, others analytically develop and infer learning rules that can elicit certain biologically inspired functions [7,8,[28][29][30].…”

Section: Related Workmentioning

confidence: 99%

A meta-learning approach to (re)discover plasticity rules that carve a desired function into a neural network

Confavreux

Agnes

Zenke

et al. 2020

Preprint

View full text Add to dashboard Cite

The search for biologically faithful synaptic plasticity rules has resulted in a large body of models. They are usually inspired by -- and fitted to -- experimental data, but they rarely produce neural dynamics that serve complex functions. These failures suggest that current plasticity models are still under-constrained by existing data. Here, we present an alternative approach that uses meta-learning to discover plausible synaptic plasticity rules. Instead of experimental data, the rules are constrained by the functions they implement and the structure they are meant to produce. Briefly, we parameterize synaptic plasticity rules by a Volterra expansion and then use supervised learning methods (gradient descent or evolutionary strategies) to minimize a problem-dependent loss function that quantifies how effectively a candidate plasticity rule transforms an initially random network into one with the desired function. We first validate our approach by re-discovering previously described plasticity rules, starting at the single-neuron level and "Oja′s rule", a simple Hebbian plasticity rule that captures the direction of most variability of inputs to a neuron (i.e., the first principal component). We expand the problem to the network level and ask the framework to find Oja′s rule together with an anti-Hebbian rule such that an initially random two-layer firing-rate network will recover several principal components of the input space after learning. Next, we move to networks of integrate-and-fire neurons with plastic inhibitory afferents. We train for rules that achieve a target firing rate by countering tuned excitation. Our algorithm discovers a specific subset of the manifold of rules that can solve this task. Our work is a proof of principle of an automated and unbiased approach to unveil synaptic plasticity rules that obey biological constraints and can solve complex functions.

show abstract

“…Research in ToDL attempts to demystify the various hidden transformations of deep architectures to provide a theoretical guarantee and understanding on the learning, approximation, optimization, and generalization capability of deep networks-and their variants-such as FNNs, convolutional neural networks (CNNs) [26], [27], recurrent neural networks (RNNs) [28], [29], autoencoders (AEs) [30]- [32], generative adversarial networks (GANs) [33]- [35], ResNet [36], and DenseNet [37], [38]. To interpret/explain learning [39], approximation [40], optimization [41], [42], and generalization [43] in these deep networks employed for classification [44], [45] and regression problems [46], advancements in ToDL have been made via numerous frameworks such as mean field theory [47]- [49], random matrix theory [50], [51], tensor factorization [52], [53], optimization theory [54]- [57], kernel learning [58]- [60], linear algebra [61], [62], spline theory [63], [64], theoretical neuroscience [65]- [67], highdimensional probability and statistics [68]- [70], manifold theory [48], [71], Fourier analysis [72], and scattering networks (vis-à-vis a wavelet transform)…”

Section: A Related Work and Motivationmentioning

confidence: 99%

“…implemented by adding the outputs of n Φ× (w,x) D,ε -fed with (w i , x i )-via an additional output layer that comprises one neuron. Accordingly, (77) where (i) follows from (67) and the assignment ε =εn; (ii) follows from these assignments:…”

Section: Appendix G Proof Of Theoremmentioning

confidence: 99%

See 1 more Smart Citation

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

Getu¹

2020

Preprint

View full text Add to dashboard Cite

Abstract—Inspired by the depth and breadth of developments on the theory of deep learning, we pose these fundamental questions: can we accurately approximate an arbitrary matrix-vector product using deep rectified linear unit (ReLU) feedforward neural networks (FNNs)? If so, can we bound the resulting approximation error? Attempting to answer these questions, we derive error bounds in Lebesgue and Sobolev norms for a matrix-vector product approximation with deep ReLU FNNs. Since a matrix-vector product models several problems in wireless communications and signal processing; network science and graph signal processing; and network neuroscience and brain physics, we discuss various applications that are motivated by an accurate matrix-vector product approximation with deep ReLU FNNs. Toward this end, the derived error bounds offer a theoretical insight and guarantee in the development of algorithms based on deep ReLU FNNs. <br>

show abstract

Neuroscience-Inspired Online Unsupervised Learning Algorithms: Artificial Neural Networks

Cited by 43 publications

References 36 publications

Habituation as a neural algorithm for online odor discrimination

Habituation as a neural algorithm for online odor discrimination

A meta-learning approach to (re)discover plasticity rules that carve a desired function into a neural network

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

Contact Info

Product

Resources

About