Why and how the brain weights contributions from a mixture of experts

O’Doherty, John P.; Lee, Sang Wan; Tadayonnejad, Reza; Cockburn, Jeff; Iigaya, Kyo; Charpentier, Caroline J.

doi:10.1016/j.neubiorev.2020.10.022

Cited by 40 publications

(46 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although the above PE minimization problem is well defined in a single context, it does not generalize to other contexts because of the tradeoff between bias and variance error (Abu-Mostafa et al, 2012;Geman et al, 1992). For example, a context-sensitive learning strategy, such as MB learning, would be suitable for minimizing the bias error, but oversensitivity inevitably accompanies variance error (Dorfman and Gershman, 2019;Filipowicz et al, 2020a;Glaze et al, 2018;Kool et al, 2017;O'Doherty et al, 2021). On the contrary, the learner could minimize the variance error by using a learning strategy that is less sensitive to context changes, such as MF learning.…”

Section: Strategy Control In Multiple Contextsmentioning

confidence: 99%

“…On the contrary, the learner could minimize the variance error by using a learning strategy that is less sensitive to context changes, such as MF learning. However, low sensitivity increases the risk of biased prediction (Dorfman and Gershman, 2019;Filipowicz et al, 2020b;Glaze et al, 2018;Kool et al, 2017;O'Doherty et al, 2021;Schulman et al, 2015).…”

Section: Strategy Control In Multiple Contextsmentioning

confidence: 99%

“…Mounting evidence supports the strategy control, concerning that the brain adjusts the balance of control between different MB and MF learning (Bogdanov et al, 2018) (adjusting w of (2)). Importantly, the lateral prefrontal cortex was found to allocate the control weights to MB and MF learning (Bogdanov et al, 2018;Lee et al, 2014;Weissengruber et al, 2019), as well as other types of learning strategies (Charpentier et al, 2020;Miyamoto et al, 2021;O'Doherty et al, 2021).…”

Section: Full Details About Adaptive Control Hypothesismentioning

confidence: 99%

See 2 more Smart Citations

Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2021

Self Cite

View full text Add to dashboard Cite

Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide modelbased and model-free reinforcement learning.

show abstract

Section: Strategy Control In Multiple Contextsmentioning

confidence: 99%

Section: Strategy Control In Multiple Contextsmentioning

confidence: 99%

Section: Full Details About Adaptive Control Hypothesismentioning

confidence: 99%

See 1 more Smart Citation

Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…If humans follow the MIX algorithm for learning and action selection, they are not supposed to reason in terms of EU or subjective values that are inexorably linked to multiplicative integration strategies. Alternatively, humans may continue computing them while processing outcomes and actions (for instance, tracking them as metacognitive landmarks) but act in accordance with the MIX algorithm and switch to EU (or subjective values) only when the reliability of the latter is high enough (suggesting a dynamic switching between computational models (Steyvers et al, 2009;Rushworth et al, 2012;Wan Lee et al, 2014;O'Doherty et al, 2021). Both cases restrain us from the direct use of behavioral measures inherent to the expected utility or subjective valuation frameworks.…”

Section: Conceptual and Inferential Issuesmentioning

confidence: 99%

Additively Combining Utilities and Beliefs: Research Gaps and Algorithmic Developments

et al. 2021

View full text Add to dashboard Cite

Value-based decision making in complex environments, such as those with uncertain and volatile mapping of reward probabilities onto options, may engender computational strategies that are not necessarily optimal in terms of normative frameworks but may ensure effective learning and behavioral flexibility in conditions of limited neural computational resources. In this article, we review a suboptimal strategy – additively combining reward magnitude and reward probability attributes of options for value-based decision making. In addition, we present computational intricacies of a recently developed model (named MIX model) representing an algorithmic implementation of the additive strategy in sequential decision-making with two options. We also discuss its opportunities; and conceptual, inferential, and generalization issues. Furthermore, we suggest future studies that will reveal the potential and serve the further development of the MIX model as a general model of value-based choice making.

show abstract

“…On the other hand, in environments with high perceived uncertainty, model-free learning are preferred over model-based learning because they are less susceptible to environmental uncertainty [ 8 ]. Accumulating evidence suggests that a part of the prefrontal cortex implements meta-control of various learning strategies, which provides a cost-effective solution to environmental uncertainty [ 1 , 9 , 10 ]. Ultimately, computational models of the brain’s meta-control principle should find a way to efficiently avoid complications arising from environmental variability.…”

Section: Introductionmentioning

confidence: 99%

Meta-control of social learning strategies

et al. 2022

Self Cite

View full text Add to dashboard Cite

Social learning, copying other’s behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others’ learning as an external knowledge base.

show abstract

Why and how the brain weights contributions from a mixture of experts

Cited by 40 publications

References 54 publications

Prefrontal solution to the bias-variance tradeoff during reinforcement learning

Prefrontal solution to the bias-variance tradeoff during reinforcement learning

Additively Combining Utilities and Beliefs: Research Gaps and Algorithmic Developments

Meta-control of social learning strategies

Contact Info

Product

Resources

About