How do real animals account for the passage of time during associative learning?

Namboodiri, Vijay Mohan K.

doi:10.1037/bne0000516

Cited by 18 publications

(18 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Collectively, we provide a new framework for understanding dopamine mediated cue-reward learning 3 that explains temporal scaling in both dopaminergic and behavioral learning (Fig 3). As learning updates occur at every reward, our model does not rely on the concept of an experimenter defined trial, which leads to many problematic assumptions 79 . Thus, it is uniquely suited to account for experience outside an arbitrarily imposed "trial period".…”

Section: Discussionmentioning

confidence: 99%

Reward timescale controls the rate of behavioral and dopaminergic learning

Burke

Jeong

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

How do we learn associations in the world (e.g., between cues and rewards)? Cue-reward associative learning is controlled in the brain by mesolimbic dopamine. It is widely believed that dopamine drives such learning by conveying a reward prediction error (RPE) in accordance with temporal difference reinforcement learning (TDRL) algorithms. TDRL implementations are trial-based: learning progresses sequentially across individual cue-outcome experiences. Accordingly, a foundational assumption, often considered a mere truism, is that the more cue-reward pairings one experiences, the more one learns this association. Here, we disprove this assumption, thereby falsifying a foundational principle of trial-based learning algorithms. Specifically, when a group of head-fixed mice received ten times fewer experiences over the same total time as another, a single experience produced as much learning as ten experiences in the other group. This quantitative scaling also holds for mesolimbic dopaminergic learning, with the increase in learning rate being so high that the group with fewer experiences exhibits dopaminergic learning in as few as four cue-reward experiences and behavioral learning in nine. An algorithm implementing reward-triggered retrospective learning explains these findings. The temporal scaling and few-shot learning observed here fundamentally changes our understanding of the neural algorithms of associative learning.

show abstract

Section: Discussionmentioning

confidence: 99%

Reward timescale controls the rate of behavioral and dopaminergic learning

Burke

Jeong

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We developed this algorithm to address problematic temporal assumptions that are foundational to common conceptions of TDRL, which result in a nonscalable representation of time ( 21 ). We tested whether this new algorithm learns causal relationships without loss of generality across timescales.…”

Section: A Retrospective Causal Learning Algorithmmentioning

confidence: 99%

Mesolimbic dopamine release conveys causal associations

Jeong

Taylor

Floeder

et al. 2022

Science

Self Cite

122

133

View full text Add to dashboard Cite

Learning to predict rewards based on environmental cues is essential for survival. It is believed that animals learn to predict rewards by updating predictions whenever the outcome deviates from expectations, and that such reward prediction errors (RPEs) are signaled by the mesolimbic dopamine system—a key controller of learning. However, instead of learning prospective predictions from RPEs, animals can infer predictions by learning the retrospective cause of rewards. Hence, whether mesolimbic dopamine instead conveys a causal associative signal that sometimes resembles RPE remains unknown. We developed an algorithm for retrospective causal learning and found that mesolimbic dopamine release conveys causal associations but not RPE, thereby challenging the dominant theory of reward learning. Our results reshape the conceptual and biological framework for associative learning.

show abstract

“…Traditional models of how animals perform trace conditioning tasks like the ones we consider here make a variety of implicit assumptions about how animals represent the passage of time [32,33]. For example, the both of which are standard assumptions of microstate representations [34].…”

Section: Discussionmentioning

confidence: 99%

Emergence of belief-like representations through reinforcement learning

Hennig

Pinto

Yamaguchi

et al. 2023

Preprint

View full text Add to dashboard Cite

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming "beliefs"---optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN's learned representation encodes belief information, but only when the RNN's capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

show abstract

How do real animals account for the passage of time during associative learning?

Cited by 18 publications

References 78 publications

Reward timescale controls the rate of behavioral and dopaminergic learning

Reward timescale controls the rate of behavioral and dopaminergic learning

Mesolimbic dopamine release conveys causal associations

Emergence of belief-like representations through reinforcement learning

Contact Info

Product

Resources

About