It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design, computational modelling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex.

No abstract

A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, another key variable has remained relatively unexplored: the role of task complexity. By using a combination of novel task design, computational modeling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process between model-based and model-free RL. We found evidence to suggest that task complexity plays a role in influencing the arbitration process alongside state-space uncertainty. Participants tended to increase modelbased RL control in response to increasing task complexity. However, they resorted to modelfree RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex bilaterally. These findings provide insight into how the inferior prefrontal cortex negotiates the trade-off between model-based and model-free RL in the presence of uncertainty and complexity, and more generally, illustrates how the brain resolves uncertainty and complexity in dynamically changing environment. SUMMARY OF FINDINGS-Elucidated the role of state-space uncertainty and complexity in model-based and model-free RL.-Found behavioral and neural evidence for complexity-sensitive prefrontal arbitration.-High task complexity induces explorative model-based RL.prompted interest in elucidating how it is the trade-off between these systems is actually managed in the brain.One influential proposal is that there exists an arbitration process that allocates control to the two systems according to various criteria (Daw et al., 2005;Kool et al., 2017;Pezzulo et al., 2018).One variant of this theory suggests that estimates about the uncertainty in the predictions of the two systems mediates the trade-off between the respective controllers, such that under situations where the model-free system has unreliable predictions, the model-based system is assigned greater weight over behavior, while under situations where the model-free system has more

We apply efficient coding principles to derive the optimal population of neurons to encode rewards from a distribution. Similar to this optimal population, dopaminergic reward prediction error neurons have a broad distribution of optimistically placed thresholds, neurons with higher thresholds have higher gain and the curvature of their responses depends on the threshold. Thus, these neurons may broadcast an efficient reward signal, not necessarily a reward prediction error.