Cognitive components underpinning the development of model-based learning

Abstract

Reinforcement learning theory distinguishes “model-free” learning, which fosters reflexive repetition of previously rewarded actions, from “model-based” learning, which recruits a mental model of the environment to flexibly select goal-directed actions. Whereas model-free learning is evident across development, recruitment of model-based learning appears to increase with age. However, the cognitive processes underlying the development of model-based learning remain poorly characterized. Here, we examined whether age-related differences in cognitive processes underlying the construction and flexible recruitment of mental models predict developmental increases in model-based choice. In a cohort of participants aged 9–25, we examined whether the abilities to infer sequential regularities in the environment (“statistical learning”), maintain information in an active state (“working memory”) and integrate distant concepts to solve problems (“fluid reasoning”) predicted age-related improvements in model-based choice. We found that age-related improvements in statistical learning performance did not mediate the relationship between age and model-based choice. Ceiling performance on our working memory assay prevented examination of its contribution to model-based learning. However, age-related improvements in fluid reasoning statistically mediated the developmental increase in the recruitment of a model-based strategy. These findings suggest that gradual development of fluid reasoning may be a critical component process underlying the emergence of model-based learning.

Increased Goal Tracking in Adolescent Rats Is Goal-Directed and Not Habit-Like

“…A substantial body of literature supports the notion that sign tracking and goal tracking are the behavioral outputs of two parallel reinforcement learning processes-akin to model-free and model-based learning, respectively (Clark et al, 2012;Huys et al, 2014). Although there has been limited investigation of model-free vs. model-based learning in adolescents, especially in animal models, the predominant view is that model-free learning is present throughout the lifespan, whereas modelbased learning emerges slowly and is not fully integrated into behavior until adulthood (Decker et al, 2016;Potter et al, 2017). Our finding that adolescent rats perform more goal tracking and less sign tracking-and that such goal tracking is truly goal-oriented behavior, based on its sensitivity to reward devaluation-complicates this view.…”

Distentangling the systems contributing to changes in learning during adolescence

“…The use of WM to manipulate information may develop over adolescence ((Crone et al, 2006)), while the use of WM to maintain information perhaps develops earlier. Recent results with a simple WM assay, a verbal span task, found age groups aged 9 to 25 performed equally well ((Potter et al, 2017)) and thus found no developmental effects on WM. The lack of age effect on the WM capacity in our study may reflect the fact that WM is used to maintain stimulus-action associations, rather than to manipulate information.…”

“…In this study, we examined developmental changes in the working memory (WM) and reinforcement learning (RL) processes that contribute to simple stimulus-action association learning. While many developmental models emphasize late prefrontal cortex maturation and gains in WM in late adolescence ((Huizinga et al, 2006;Larsen & Luna, 2018)), previous studies have also shown changes in development in reinforcement learning tasks (Palminteri et al, 2016;Potter et al, 2017;Van Den Bos et al, 2012).…”

“…Adolescence is widely associated with the development of higher cognitive resources and the use of increasingly complex learning strategies ((Crone et al, 2006;Decker et al, 2016;Potter et al, 2017;Selmeczy et al, 2018)). These shifts in learning are mirrored by the development of prefrontal cortex, which does not reach functional or structural maturation until late in the second decade or even middle of the third decade of life ((Bunge & Gabrieli, 2002;Casey, Jones, & Hare, 2008)).…”

“…To understand the effects of development on behavior and on model parameters, we ran a series of analyses at group and individual levels. First, following the practice in the literature to separate "children" from "adolescents" and adults (Potter, Bryce, & Hartley, 2017) we grouped 8 to 12, 13 to 17, and 25 to 30-year-olds into separate groups and ran one-way ANOVAs on behavioral and modeling measures of interest, looking for broad group-based, age-related differences in cognition; ANOVAs were replaced by Kruskal-Wallis non-parametric tests for non-normal measurements. Where group effects were present, we ran post-hoc tests to identify which group drove the effect.…”