A multiplicative reinforcement learning model capturing learning dynamics and interindividual variability in mice

Abstract

Both in humans and in animals, different individuals may learn the same task with strikingly different speeds; however, the sources of this variability remain elusive. In standard learning models, interindividual variability is often explained by variations of the learning rate, a parameter indicating how much synapses are updated on each learning event. Here, we theoretically show that the initial connectivity between the neurons involved in learning a task is also a strong determinant of how quickly the task is learned, provided that connections are updated in a multiplicative manner. To experimentally test this idea, we trained mice to perform an auditory Go/NoGo discrimination task followed by a reversal to compare learning speed when starting from naive or already trained synaptic connections. All mice learned the initial task, but often displayed sigmoid-like learning curves, with a variable delay period followed by a steep increase in performance, as often observed in operant conditioning. For all mice, learning was much faster in the subsequent reversal training. An accurate fit of all learning curves could be obtained with a reinforcement learning model endowed with a multiplicative learning rule, but not with an additive rule. Surprisingly, the multiplicative model could explain a large fraction of the interindividual variability by variations in the initial synaptic weights. Altogether, these results demonstrate the power of multiplicative learning rules to account for the full dynamics of biological learning and suggest an important role of initial wiring in the brain for predispositions to different tasks.behavior | memory | cue competition | savings I t is commonly observed in animal behavior experiments, as much as in the classroom, that different individuals eventually learn the same task with strikingly different dynamics. Many factors were shown to influence learning speed and/or performance at the group level, including genetic background (1), early experience (2), or contextual factors such as stress (3). In all these cases, the underlying idea is that these factors act on synaptic plasticity mechanisms to change different parameters that modulate or gate synaptic updates, thereby modifying learning dynamics at the system scale (e.g., ref. 4). These ideas are in line with most theoretical models of biological learning (5), such as reinforcement learning (6, 7), which are based on the trial-by-trial update of mathematical variables that can be directly or indirectly related to updates of synaptic weights between neurons. In these models, one essential parameter controlling the speed of learning is the learning rate that scales the learning rule (i.e., the rule according to which synapses are updated). Other factors are also known to have an influence on learning speed such as the level of noise and the particular learning rule (8) or metaparameters that dynamically control the balance between different aspects of learning behavior like the exploration vs. exploitation or memory storage vs. ren...

Cortical recruitment determines learning dynamics and strategy

“…3b), thereby demonstrating their ability to discriminate the two sounds. Importantly, as typically observed in such tasks 24 , the S+ sound was rapidly associated with the lick response and the rate limiting factor in the acquisition of the task was to associate the suppression of licking with the S-sound ( Fig. 3b).…”

“…Using auditory discrimination tasks of sounds with different global cortical response strengths, , we show that cortical recruitment impacts learning dynamics 23,24 in a manner similar to the salience parameter of a reinforcement learning model. To explore this result in more precise experimental settings, we trained mice to discriminate optogenetically-driven response patterns that elicit different levels of cortical activity.…”

Rethinking dopamine as generalized prediction error

“…There is a large body of evidence in associative learning suggesting an imbalance between excitatory and inhibitory learning [87,88]. Mirroring this imbalance is an asymmetry in the dynamic range of the firing rate of single dopaminergic neurons in the midbrain [2].…”

Reversal Learning in Humans and Gerbils: Dynamic Control Network Facilitates Learning

“…This increased efficiency of performance in serial reversal tasks cannot be explained by learning mechanisms which are solely based on the initially formed associations of complex stimulus features to categorical responses, since this leads to decreased efficiency due to interference effects from the previously learned contingencies (Pubols, 1957; Clayton, 1962; Gossette and Inman, 1966; Feldman, 1968; Gossette and Hood, 1968; Kulig and Calhoun, 1972; Garner et al, 1996; Bathellier et al, 2013; Kangas and Bergman, 2014). …”