Separating Probability and Reversal Learning in a Novel Probabilistic Reversal Learning Task for Mice

Abstract

The exploration/exploitation tradeoff-pursuing a known reward vs. sampling from lesser known options in the hope of finding a better payoff-is a fundamental aspect of learning and decision making. In humans, this has been studied using multi-armed bandit tasks. The same processes have also been studied using simplified probabilistic reversal learning (PRL) tasks with binary choices. Our investigations suggest that protocols previously used to explore PRL in mice may prove beyond their cognitive capacities, with animals performing at a no-better-than-chance level. We sought a novel probabilistic learning task to improve behavioral responding in mice, whilst allowing the investigation of the exploration/exploitation tradeoff in decision making. To achieve this, we developed a two-lever operant chamber task with levers corresponding to different probabilities (high/low) of receiving a saccharin reward, reversing the reward contingencies associated with levers once animals reached a threshold of 80% responding at the high rewarding lever. We found that, unlike in existing PRL tasks, mice are able to learn and behave near optimally with 80% high/20% low reward probabilities. Altering the reward contingencies towards equality showed that some mice displayed preference for the high rewarding lever with probabilities as close as 60% high/40% low. Additionally, we show that animal choice behavior can be effectively modelled using reinforcement learning (RL) models incorporating learning rates for positive and negative prediction error, a perseveration parameter, and a noise parameter. This new decision task, coupled with RL analyses, advances access to investigate the neuroscience of the exploration/exploitation tradeoff in decision making.

Reinforcement Learning and Bayesian Inference Provide Complementary Models for the Unique Advantage of Adolescents in Stochastic Reversal

“…RL is a popular framework to model probabilistic reversal learning (Boehme et al, 2017;Chase et al, 2010;Gläscher et al, 2009;Hauser et al, 2015;Javadi et al, 2014;Metha et al, 2020;Peterson et al, 2009). RL agents choose actions based on action values that reflect actions' expected long-term cumulative reward.…”

“…Such model-independent analyses have led to many interesting insights, but have been unable to test hypotheses about specific cognitive processes that are at work while subjects perform the task. In an effort to better understand these processes, more recent studies have started to employ computational modeling, most often in the RL framework (e.g., Boehme et al, 2017; Chase et al, 2010; Gläscher et al, 2009; Hauser et al, 2015; Javadi et al, 2014; Metha et al, 2020; Peterson et al, 2009).…”

CADM2is implicated in impulsive personality and numerous other traits by genome- and phenome-wide association studies in humans and mice

“…Because Cadm2 has been previously associated with alcohol consumption, we used a within-subjects design to assess risky behavior under acute doses of ethanol (0, 0.5, 1g/kg), as previously described (41). In addition, we evaluated motivation, as measured by a Progressive Ratio Breakpoint task [ PBRT , (42, 43)], and behavioral flexibility, as measured by a Probabilistic Reversal Learning task [ PRL , (44)]. General exploration was measured via the Behavioral pattern monitor [ BPM , (45, 46)].…”

Noradrenergic Regulation of Two-Armed Bandit Performance

“…Other studies have thus referred to task design as a "bandit" task--a reinforcement learning paradigm. Within this framework, the focus is on changes in two variables: learning rate, which describes how quickly the subject incorporate new evidence into its decisions, and inverse temperature, which describes the subject's confidence in the decision (Groman et al, 2016;Metha et al, 2019;Sutton & Barto, 2018). These two approaches, WSLS and reinforcement learning, are rarely combined to understand behavioral mechanisms or neuronal measures of reversal learning (Worthy & Maddox, 2014).…”

“…Probabilistic outcomes that force animals to integrate multiple previous trials to guide choice increase the difficulty of the task. Other studies have thus referred to probabilistic reversal learning as a “bandit” task--a reinforcement learning paradigm (Groman et al, 2016; Metha et al, 2019; Sutton & Barto, 2018). These two approaches are rarely combined to understand behavioral mechanisms or neuronal measures of reversal learning (Worthy & Maddox, 2014).…”