Quantifying exploration in reward-based motor learning

Mastrigt, None van; Smeets, Jeroen B. J.; Kooij, Katinka van der

doi:10.1101/869735

Cited by 9 publications

(12 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A variety of studies show that reinforcement learning, or instrumental conditioning, can learn to compensate for a visuomotor perturbations (Butcher et al 2017; Cashaback et al 2017, 2019; Codol et al 2018; Darshan et al 2014; Holland et al 2018; Izawa and Shadmehr 2011; Mastrigt et al 2020; Palidis et al 2019; Pekny et al 2015; Therrien et al 2015). For instance, in such studies, participants are required to reach to a target without visual feedback relating to their hand position during or after their movements.…”

Section: Adaptation Consists Of Distinct Processes Driven By Different Errorsmentioning

confidence: 99%

A Synthesis of the Many Errors and Learning Processes of Visuomotor Adaptation

Morehead

Xivry

2021

Preprint

View full text Add to dashboard Cite

Visuomotor adaptation has one of the oldest experimental histories in psychology and neuroscience, yet its precise nature has always been a topic of debate. Here we offer a survey and synthesis of recent work on visuomotor adaptation that we hope will prove illuminating for this ongoing dialogue. We discuss three types of error signals that drive learning in adaptation tasks: task performance error, sensory prediction-error, and a binary target hitting error. Each of these errors has been shown to drive distinct learning processes. Namely, both target hitting errors and putative sensory prediction-errors drive an implicit change in visuomotor maps, while task performance error drives learning of explicit strategy use and non-motor decision-making. Each of these learning processes contributes to the overall learning that takes place in visuomotor adaptation tasks, and although the learning processes and error signals are independent, they interact in a complex manner. We outline many task contexts where the operation of these processes is counter-intuitive and offer general guidelines for their control, measurement and interpretation. We believe this new framework unifies several disparate threads of research in sensorimotor adaptation that often seem in conflict. We conclude by explaining how this more nuanced understanding of errors and learning processes could lend itself to the analysis of other types of sensorimotor adaptation, of motor skill learning, of the neural processing underlying sensorimotor adaptation in humans, of animal models and of brain computer interfaces.

show abstract

Section: Adaptation Consists Of Distinct Processes Driven By Different Errorsmentioning

confidence: 99%

A Synthesis of the Many Errors and Learning Processes of Visuomotor Adaptation

Morehead

Xivry

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In wildtype mice, improved performance of the task was primarily driven by a decrease in the inter-trial variability of pull kinematics across days of testing. Variability during the early stages of reinforcement learning is advantageous, as exploration is necessary in order to determine execution parameters that will lead to reward (Dhawale et al, 2017;Van Mastrigt et al, 2020). However, as information about the outcomes of different movements becomes available, variability is reduced in order to improve performance (Pekny et al, 2015;Dhawale et al, 2019).…”

Section: Discussionmentioning

confidence: 99%

Impaired refinement of kinematic variability in Huntington disease mice on an automated home-cage forelimb motor task

Woodard

Sepers

Raymond

2021

Preprint

View full text Add to dashboard Cite

The effective development of novel therapies in mouse models of neurological disorders relies on behavioural assessments that provide accurate read-outs of neuronal dysfunction and/or degeneration. We designed an automated behavioural testing system (‘PiPaw’) which integrates an operant lever-pulling task directly into the mouse home-cage. This task is accessible to group-housed mice 24-hours per day, enabling high-throughput longitudinal analysis of forelimb motor learning. Moreover, this design eliminates the need for exposure to novel environments and minimizes experimenter interaction, significantly reducing two of the largest stressors associated with animal behaviour. Mice improved their performance of this task over one week of testing by reducing inter-trial variability of reward-related kinematic parameters (pull amplitude or peak velocity). In addition, mice displayed short-term improvements in reward rate, and a concomitant decrease in movement variability, over the course of brief (<10 minutes) bouts of task engagement. We used this system to assess motor learning in mouse models of the inherited neurodegenerative disorder, Huntington disease (HD). Despite having no baseline differences in task performance, Q175-FDN HD mice were unable to modulate the variability of their movements in order to increase reward on either short or long timescales. Task training was associated with a decrease in the amplitude of spontaneous excitatory activity recorded from striatal medium spiny neurons in the hemisphere contralateral to the trained forelimb in wildtype mice; however, no such changes were observed in Q175-FDN mice. This behavioural screening platform should prove useful for preclinical drug trials towards improved treatments in HD and other neurological disorders.Significance StatementIn order to develop effective therapies for neurological disorders such as Huntington disease (HD), it’s important to be able to accurately and reliably assess the behaviour of mouse models of these conditions. Moreover, these behavioural assessments should provide an accurate readout of underlying neuronal dysfunction and/or degeneration. In this paper, we employed an automated behavioural testing system to assess motor learning in mice within their home-cage. Using this system, we were able to study motor abnormalities in HD mice with an unprecedented level of detail, and identified a specific behavioural deficit associated with an underlying impairment in striatal neuronal plasticity. These results validate the usefulness of this system for assessing behaviour in mouse models of HD and other neurological disorders.

show abstract

“…While changes in TTV after S-trials can be due to sensorimotor noise and active exploration, changes in TTV after S+ trials should be mainly caused by sensorimotor noise (van Mastrigt et al, 2020). Thus, motor variability is influenced by a number of factors including intended and unintended variability regulatory mechanisms (Pekny et al, 2015;Therrien et al, 2018;Dhawale et al, 2019;van Mastrigt et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

Behavioural variability and cortical electrophysiological signals depend on recent outcomes during human reinforcement motor learning

Wiegel

Spedden

Ramsenthaler

et al. 2021

Preprint

View full text Add to dashboard Cite

The history of our actions and the outcomes of these represent important information, which can inform choices, and efficiently guide future behaviour. While unsuccessful (S-) outcomes are expected to lead to more explorative motor states and increased behavioural variability, successful (S+) outcomes lead to reinforcement of the previous action and thus exploitation. Here, we show that during reinforcement motor learning, humans attribute different values to previous actions when they experience S- vs. S+ outcomes. Behavioural variability after S- outcomes is influenced more by the previous outcomes compared to what is observed after S+ outcomes. Using electroencephalography, we show that neural oscillations of the prefrontal cortex encode the level of reinforcement (high beta frequencies) and reflect the detection of reward prediction errors (theta frequencies). The results suggest that S+ experiences 'overwrite' previous motor states to a greater extent than S- experiences and that modulations in neural oscillations in the prefrontal cortex play a potential role in encoding the (changes in) movement variability state during reinforcement motor learning.

show abstract

Quantifying exploration in reward-based motor learning

Cited by 9 publications

References 39 publications

A Synthesis of the Many Errors and Learning Processes of Visuomotor Adaptation

A Synthesis of the Many Errors and Learning Processes of Visuomotor Adaptation

Impaired refinement of kinematic variability in Huntington disease mice on an automated home-cage forelimb motor task

Behavioural variability and cortical electrophysiological signals depend on recent outcomes during human reinforcement motor learning

Contact Info

Product

Resources

About