Modeling changes in probabilistic reinforcement learning during adolescence

Abstract

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Confirmatory reinforcement learning changes with age during adolescence

“…In particular, adolescence, defined as the developmental period between 10 and 24 (Sawyer et al 2018), is a stage of life characterised by changes in exploration, learning and decision-making. In line with this, studies investigating the development of reinforcement learning suggest that adults are less exploratory and often more accurate in their reinforcement learning behaviour than adolescents between 12 and 18 years (Christakou et al, 2013;Decker et al, 2016;Javadi et al, 2014;Lloyd et al, 2021;Palminteri et al, 2016;Rodriguez Buritica et al, 2019;Xia et al, 2021-see Bolenz et al, 2017and Nussenbaum & Hartley, 2019. This raises the question of whether such age-related differences in reinforcement learning could be explained by an emerging confirmation bias.…”

“…For the standard confirmation model (CM) we considered 4 variants: CM αDis (i.e., in which only α Dis is fixed but α Con and β are allowed to vary), CM β (in which only β is fixed and both learning rates are allowed to vary), CM αCon, αDis (in which both learning rates are fixed and only β varies) and CM αDis, β (in which α Dis and β are fixed and only α Con is allowed to vary). We did not consider CM alternatives emphasising variance in α Dis , as variance in this learning rate is frequently negligible (Chambon et al, 2020;Palminteri et al, 2017;Schüller et al, 2020;Xia et al, 2021).…”

The Interpretation of Computational Model Parameters Depends on the Context

“…This section gives a brief overview of the experimental tasks and computational models (details in Fig. 1C-E, sections 4.4 and 4.5, and original publications [33,34,35]), before showing the results on parameter generalizability (section 2.1) and interpretability (section 2.2).…”

“…To illustrate this point, the current project started out as an investigation into the adolescent development of learning, with the aim of combining the insights of three different learning tasks to gain a more complete understanding of adolescent learning. However, we soon realized that even though each task individually showed strong and interesting developmental patterns [33,34,35], these patterns were very different-and even contradictoryacross tasks. This showed us that specific model parameters (e.g., learning rate) did not isolate-and measure individuals on-specific cognitive processes (e.g., value updating) that were distinct and theoretically separate from the cognitive processes (e.g., decision making) identified by other parameters (e.g., decision temperature), as we had implicitly assumed.…”

“…The three learning tasks varied on several common dimensions, including feedback stochasticity, task volatility, and memory demands (Fig. 1B), and have been used previously to study RL processes [33,34,35] (However, like many tasks in the literature, these tasks likely also engaged other cognitive processes besides RL, such as working memory and reasoning.) The within-participant design allowed us to test directly whether the same participants showed the same parameters across tasks (generalizability), and the combination of multiple tasks shed light on which cognitive processes the same parameters captured in each task (interpretability).…”