Prefrontal solution to the bias-variance tradeoff during reinforcement learning

Abstract

Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide modelbased and model-free reinforcement learning.

Long short-term prediction guides human metacognitive reinforcement learning

“…Recent evidence has shown that the human brain weights these strategies according to context changes ( [23,[23][24][25][26][27][28][29]), resonating with the long-term prediction view. This evidence raised a new possibility that the brain implements meta-learning [15] or a meta-control of multiple learning strategies [30].…”

“…The adaptability test simulates a highly volatile environment, for which we developed two scenarios: the one with sudden task changes randomly sampled from the set of 10 tasks (Figure 3D -task switching scenario) and the other with continuous, subtle task parameter changes in a random walk fashion within a single task (Figure 3E -context changing scenario). A choice optimality measure was used in the latter scenario because a simple performance metric, such as average or cumulative reward, is not sensitive enough to evaluate adaptive behavior to subtle context changes [25].…”

“…For prefrontal RL, we used the computational model adaptively combining MB and ,MF control, inspired from the findings on the neural activity of the lateral prefrontal cortex and ventral striatum [23,24]. Two versions were selected: a baseline model for pfcRL1 [23] and an adaptive model for pfcRL2 [25]. Both models successfully implement arbitration control based on the Bayesian reliability estimation over MB RL (FORWARD learning [20]) and MF RL (SARSA Long short-term prediction guides human metacognitive reinforcement learning learning [51]).…”

“…Choice optimality is a binary value (i.e. 1 or 0) that is the proximity of an agent's choices to the optimal agent's choices, and is often used to assess the degree of engagement of MB in the context of the arbitration control between MB and MF [25,55]. It also can be defined in terms of mutual information (See Episodic encoding efficacy test via information theoretic analysis).…”

Neurocomputational model of compulsivity: deviating from an uncertain goal-directed system

“… 19 , 24 , 25 When the goal-directed system is deemed to be dominant over the other, the prefrontal arbitrator suppresses the posterior putamen, which is a habitual system controller involved in the estimation of the prediction error or action value of the system. 19 , 26 Because OCD is characterized by enhanced intolerance of uncertainty regarding changes in goal-directed contingency, 27 , 28 this uncertainty-based arbitration model would fit to test the imbalance theory in OCD. Given the crucial role of uncertainty differences in the arbitration process, the conditions of the Markov decision task provide a suitable environment for inducing transitions between the strategies by manipulating their prediction error or uncertainty.…”