Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales

Abstract

Cognitive control is typically understood as a set of mechanisms which enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine beheavior or making choices beneficial only at a short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive, the specific question of how uncertain consequences at different time scales are integrated into adaptive decisions remains poorly understood. Here, we propose that precisely the problem of integrating experience and forming elaborate plans over multiple time scales is a key component for better understanding how human agents solve cognitive control dilemmas such as the exploration-exploitation dilemma. In support of this conjecture, we present a computational model of probabilistic inference over hidden states and actions, which are represented as a hierarchy of time scales. Simulations of goal-reaching agents instantiating the model in an uncertain and dynamic task environment show how the exploration-exploitation dilemma may be solved by inferring meta-control states which adapt behaviour to changing contexts.

Transfer of learned cognitive flexibility to novel stimuli and task sets

“…A couple of studies that examined the influence of forced-choice task switching on voluntary task switches, found that increasing the proportion of forced choices, in particular in combination with high switch rates, increases voluntary task switching rates (Chiu et al, 2020;Fröber & Dreisbach, 2017). Taken together, this suggests that in conditions where experience is used to learn task models (Niv, 2019), meta-flexibility may be achieved by altering one's updating threshold or the rate at which this threshold is reached (both would be observed as a change in learning rate in RL modeling), which in turn may promote transferable effects (Goschke, 2003;Marković et al, 2019), as observed in the present study.…”

The exploration-exploitation trade-off in a foraging task is affected by mood-related arousal and valence

“…Indeed, previous work within the domain of cognitive psychology has not drawn a sharp line between Online Supplementary Material can be found at: https://osf.io/bm6ps/. We made our data, material, and code (licensed under a Creative Commons Attribution-NonCommercial 4.0 International License) available at: https://osf.io/skh5y/ exploration-exploitation on the one hand and flexibilitypersistence on the other (Hills, 2006;Hills & Dukas, 2012;Hommel & Colzato, 2017;Marković et al, 2019). For instance, and among others, Hills et al (2008Hills et al ( , 2010 have illustrated that such theoretical overlap is not far-fetched, because abstract conceptual search can be primed by inducing explorative or exploitative foraging strategies, with high levels of exploration (vs. exploitation) boosting flexible (vs. persistent) cognitive search patterns.…”

The Willpower Paradox: Possible and Impossible Conceptions of Self-Control

“…15 Feasible solutions of such metacontrol problems must therefore rely on learning processes that enable agents to infer from prior experience in which contexts it is adaptive (and maximizes reward in the long run) to stick to prior commitments and when it is more adaptive to adjust preferences more myopically in light of one's current motivational state. Initial computational-modeling studies indicate that experience-based learning in conjunction with Bayesian inference may provide a basis for solving such metacontrol problems (Markovic et al, 2021). However, we are not aware of any attempts to date to apply such models to the domain of self-control, and it will thus be a key challenge for theories favoring Resolution 3 to apply such models to preventive self-control and test them empirically.…”