Expressivity of expand-and-sparsify representations

Dasgupta, Sanjoy; Tosh, Christopher

doi:10.48550/arxiv.2006.03741

Search citation statements

Order By: Relevance

Paper Sections

Select...

Circuit Mechanisms For Continual Learning In Fruit Flies1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2021

Publication Types

Select...

Other1

Relationship

Self Cite1

Independent0

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This transformation was previously studied in the context of similarity search [40][41][42][43], compressed sensing [35,44], and pattern separation for subsequent learning [45][46][47].…”

Section: Circuit Mechanisms For Continual Learning In Fruit Fliesmentioning

confidence: 99%

Algorithmic insights on continual learning from fruit flies

Shen¹,

Dasgupta²,

Navlakha³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Continual learning in computational systems is challenging due to catastrophic forgetting. We discovered a two-layer neural circuit in the fruit fly olfactory system that addresses this challenge by uniquely combining sparse coding and associative learning. In the first layer, odors are encoded using sparse, high-dimensional representations, which reduces memory interference by activating non-overlapping populations of neurons for different odors. In the second layer, only the synapses between odor-activated neurons and the odor's associated output neuron are modified during learning; the rest of the weights are frozen to prevent unrelated memories from being overwritten. We show empirically and analytically that this simple and light-weight algorithm significantly boosts continual learning performance. The fly's associative learning algorithm is strikingly similar to the classic perceptron learning algorithm, albeit two modifications, which we show are critical for reducing catastrophic forgetting. Overall, fruit flies evolved an efficient life-long learning algorithm, and circuit mechanisms from neuroscience can be translated to improve machine computation.

show abstract

“…This transformation was previously studied in the context of similarity search [40][41][42][43], compressed sensing [35,44], and pattern separation for subsequent learning [45][46][47].…”

Section: Circuit Mechanisms For Continual Learning In Fruit Fliesmentioning

confidence: 99%