Dopamine role in learning and action inference

Bogacz, Rafał

doi:10.7554/elife.53262

Cited by 55 publications

(41 citation statements)

References 82 publications

(147 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, the results of our three approximate methods (Particle Filtering, Variational SMiLe, and Message Passing with a fixed N number of messages) as well as some previously developed ones (Adams & MacKay, 2007;Fearnhead & Liu, 2007;Nassar et al, 2012Nassar et al, , 2010 demonstrate that the surprise-based modulation of the learning rate is a generic phenomenon. Therefore, regardless of whether the brain uses Bayesian inference or an approximate algorithm (Bogacz, 2017(Bogacz, , 2019Findling et al, 2019;Friston, 2010;Gershman, 2019;Gershman et al, 2014;Mathys et al, 2011;Nassar et al, 2012Nassar et al, , 2010Prat-Carrabin et al, 2020), the notion of Bayes Factor Surprise and the way it modulates learning (see equations 2.12 and 2.9) look generic.…”

Section: Surprise Modulation As a Generic Phenomenonmentioning

confidence: 99%

Learning in Volatile Environments With the Bayes Factor Surprise

et al. 2021

View full text Add to dashboard Cite

Surprise-based learning allows agents to rapidly adapt to nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call the Bayes Factor Surprise, that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms, the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprise-based algorithms, one in the family of particle filters, one in the family of variational learning, and one in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from the Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes Factor Surprise from the Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior and to reinforcement learning in nonstationary environments.

show abstract

Section: Surprise Modulation As a Generic Phenomenonmentioning

confidence: 99%

Learning in Volatile Environments With the Bayes Factor Surprise

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In our present formulation, the sensorimotor inference was not included; however, a mechanism of motor inference can be included explicitly by considering a Langevin equation for a sensorimotor state. This procedure extends the probabilistic generative model by accommodating the prior density for motor planning for active perception, which is similar to what was done in Bogacz ( 2020 ).…”

Section: Discussionmentioning

confidence: 99%

Bayesian mechanics of perceptual inference and motor control in the brain

Kim

2021

Biol Cybern

View full text Add to dashboard Cite

The free energy principle (FEP) in the neurosciences stipulates that all viable agents induce and minimize informational free energy in the brain to fit their environmental niche. In this study, we continue our effort to make the FEP a more physically principled formalism by implementing free energy minimization based on the principle of least action. We build a Bayesian mechanics (BM) by casting the formulation reported in the earlier publication (Kim in Neural Comput 30:2616–2659, 2018, 10.1162/neco_a_01115) to considering active inference beyond passive perception. The BM is a neural implementation of variational Bayes under the FEP in continuous time. The resulting BM is provided as an effective Hamilton’s equation of motion and subject to the control signal arising from the brain’s prediction errors at the proprioceptive level. To demonstrate the utility of our approach, we adopt a simple agent-based model and present a concrete numerical illustration of the brain performing recognition dynamics by integrating BM in neural phase space. Furthermore, we recapitulate the major theoretical architectures in the FEP by comparing our approach with the common state-space formulations.

show abstract

“…Thus, low levels of dopamine are likely to fit the ideal hard selection template, while higher levels are associated with soft selection. We consider this transition – from hard to soft selection – to be a feature of dopamine control of basal ganglia function (Blanco & Sloutsky, 2020; Bogacz, 2020; Costa, 2007; Costa et al, 2006; Gizer et al, 2009; Wickens et al, 2007). If the basal ganglia supports such a mechanism, then values of the parameters w D 1 , w D 2 which will optimise this transitional control will reflect the relative biological dopamine sensitivity these parameters aim to capture.…”

Section: Quantifying Selection Capabilitymentioning

confidence: 99%

Differential dopamine receptor-dependent sensitivity improves action selection in the basal ganglia

Codol

Gribble

2020

Preprint

View full text Add to dashboard Cite

The problem of selecting one action from a set of different possible actions, simply referred to as the problem of action selection, is a ubiquitous challenge in the animal world. For vertebrates, the basal ganglia (BG) are widely thought to implement the core computation to solve this problem, as the anatomy and physiology of the BG are well-suited to this end. However, the BG still displays physiological features whose role in achieving efficient action selection remains unclear. In particular, it is known that the two types of dopaminergic receptors (D1 and D2) present in the BG give rise to mechanistically different responses. The overall effect will be a difference in sensitivity to dopamine which may have ramifications for action selection. However, which receptor type leads to a stronger response is, a priori, unclear, due to the complexity of the intracellular mechanisms involved. In this study, we use the action selection hypothesis to {\em predict} which of D1 or D2 has the greater sensitivity. Thus, we ask - what sensitivity ratio would result in enhanced action selection functionality in the basal ganglia? To do this, we incorporated differential D1 and D2 sensitivity in an existing, high level computational model of the macro-architecture of the basal ganglia, via a simple weighting variable. We then quantitatively assessed the model's capacity to perform action selection as we parametrically manipulated the new feature. We show that differential (rather than equal) D1 and D2 sensitivity to dopaminergic input improves action selection, and specifically, that greater D1 sensitivity (compared to that for D2) leads to these improvements.

show abstract

Dopamine role in learning and action inference

Cited by 55 publications

References 82 publications

Learning in Volatile Environments With the Bayes Factor Surprise

Learning in Volatile Environments With the Bayes Factor Surprise

Bayesian mechanics of perceptual inference and motor control in the brain

Differential dopamine receptor-dependent sensitivity improves action selection in the basal ganglia

Contact Info

Product

Resources

About