Distributional reinforcement learning in prefrontal cortex

Th, Muller; Jl, Butler; Veselič, Sebastijan; Miranda, Bruno; Te, Behrens; Kurth‐Nelson, Zeb; Sw, Kennerley

doi:10.1101/2021.06.14.448422

Cited by 5 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Numerous theoretical models currently exist which attempt to provide normative descriptions of this role of the PFC, often from the perspective of reinforcement learning, meta-learning, or both (J. X. Wang et al, 2018; Muller et al, 2024).…”

Section: Discussionmentioning

confidence: 99%

A dual-receptor model of serotonergic psychedelics

Juliani,

Chelu,

Graesser

et al. 2024

Preprint

View full text Add to dashboard Cite

Serotonergic psychedelics have been identified as promising next-generation therapeutic agents in the treatment of mood and anxiety disorders. While their efficacy has been increasingly validated, the mechanism by which they exert a therapeutic effect is still debated. A popular theoretical account is that excessive 5-HT2a agonism disrupts cortical dynamics, relaxing the precision of maladaptive high-level beliefs, thus making them more malleable and open to revision. We extend this perspective by developing a theoretical framework and simulations based on predictive processing and an energy-based model of cortical dynamics. We consider the role of both 5-HT2a and 5-HT1a agonism, characterizing 5-HT2a agonism as inducing stochastic perturbations of the energy function underlying cortical dynamics and 5-HT1a agonism as inducing a global smoothing of that function. Within our simulations, we find that while both agonists are able to provide a significant therapeutic effect individually, mixed agonists provide both a more psychologically tolerable acute experience and better therapeutic efficacy than either pure 5-HT2a or 5-HT1a agonists alone. This finding provides a potential theoretical basis for the clinical success of LSD, psilocybin, and DMT, all of which are mixed serotonin agonists. Our results furthermore indicate that exploring the design space of biased 5-HT1a agonist psychedelics such as 5-MeO-DMT may prove fruitful in the development of even more effective and tolerable psychotherapeutic agents in the future.

show abstract

Section: Discussionmentioning

confidence: 99%

A dual-receptor model of serotonergic psychedelics

Juliani,

Chelu,

Graesser

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The CPD was estimated at each timepoint for each neuron within a region and the average across all neurons per region was plotted over time. Furthermore, the CPD values were averaged within a larger 300 millisecond time window locked to choice onset, corresponding to conventional time windows used previously to identify value-related representations emerging in these four regions 1,4…”

Section: Methodsmentioning

confidence: 99%

A cognitive map for value-guided choice in ventromedial prefrontal cortex

Veselic,

Muller,

Gutierrez

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

The prefrontal cortex is crucial for economic decision-making and representing the value of options. However, how such representations facilitate flexible decisions remains unknown. We reframe economic decision-making in prefrontal cortex in line with representations of structure within the medial temporal lobe because such cognitive map representations are known to facilitate flexible behaviour. Specifically, we framed choice between different options as a navigation process in value space. Here we show that choices in a 2D value space defined by reward magnitude and probability were represented with a grid-like code, analogous to that found in spatial navigation. The grid-like code was present in ventromedial prefrontal cortex (vmPFC) local field potential theta frequency and the result replicated in an independent dataset. Neurons in vmPFC similarly contained a grid-like code, in addition to encoding the linear value of the chosen option. Importantly, both signals were modulated by theta frequency – occurring at theta troughs but on separate theta cycles. Furthermore, we found sharp-wave ripples – a key neural signature of planning and flexible behaviour – in vmPFC, which were modulated by accuracy and reward. These results demonstrate that multiple cognitive map-like computations are deployed in vmPFC during economic decision-making, suggesting a new framework for the implementation of choice in prefrontal cortex.

show abstract

“…Temporal difference learning always converges to the true stimulus values; however, people often deviate from this linear learning trajectory in important ways. For example, there are asymmetric learning rates for rewarding vs. punishing stimuli (Muller et al, 2021), the value function to be learned can be non-linear and learning itself is often distributed (François-Lavet et al, 2018). Further, when the number of states (e.g., a spatial location, a physiological state of being, or a mental state of mind) is large the time it takes to learn the value function is infeasible for animals.…”

Section: Modeling Exploitationmentioning

confidence: 99%

How generalization relates to the exploration-exploitation tradeoff

Houser

2023

Front. Cognit.

View full text Add to dashboard Cite

It is known that animals foraging in the wild must balance their levels of exploitation and exploration so as to maximize resource consumption. This usually manifests as an area-restricted search strategy, such that animals tend to exploit environmental patches and make long excursions between patches. This optimal foraging strategy, however, relies on an underlying assumption: nearby locations yield similar resources. Here, we offer an explanation as to how animals utilize this assumption, which implicitly involves generalization. We also describe the computational mechanisms hypothesized to incorporate factors of exploitation, exploration, and generalization, thus, providing a more holistic picture of animal search strategies. Moreover, we connect this foraging behavior to cognition in general. As such, we suggest that cognitive processes, particularly those involved in sequential decision-making, reuse the computational principles grafted into neural activity by the evolution of optimal foraging. We speculate as to what neurobiological substrates may be using area-restricted search, as well as how a model of exploitation, exploration, and generalization can inform psychopathology.

show abstract

Distributional reinforcement learning in prefrontal cortex

Cited by 5 publications

References 37 publications

A dual-receptor model of serotonergic psychedelics

A dual-receptor model of serotonergic psychedelics

A cognitive map for value-guided choice in ventromedial prefrontal cortex

How generalization relates to the exploration-exploitation tradeoff

Contact Info

Product

Resources

About