Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise

Abstract

See Miall and Galea (doi: ) for a scientific commentary on this article. Cerebellar lesions impair both coordination and motor learning. Therrien et al. show that affected individuals can learn using a reinforcement mechanism despite a deficit in error-based motor learning. They also identify a critical feature of cerebellar patients’ movements (motor noise), which determines the effectiveness of learning under reinforcement.

Exploration of joint redundancy but not task space variability facilitates supervised motor learning

“…Although minimizing variability is expected to improve taskrelated performance, recent evidence suggests that motor variability paradoxically helps in motor learning (16)(17)(18)(19)(20). Such findings are supported by ideas in reinforcement learning (21,22), which suggest that baseline variability increases exploration, which in turn facilitates learning.…”

“…Consistent with this view, it has been observed in a wide range of tasks (2-7) that variability is not eliminated but optimized (8-11) to accumulate in a task-relevant dimensions using a minimum-intervention principle (12). Such variability, that is a consequence of redundancy, can be quantified as an uncontrolled manifold (13-15) in which task-independent variability is constrained to a redundant subspace (or "uncontrolled manifold").Although minimizing variability is expected to improve taskrelated performance, recent evidence suggests that motor variability paradoxically helps in motor learning (16)(17)(18)(19)(20). Such findings are supported by ideas in reinforcement learning (21,22), which suggest that baseline variability increases exploration, which in turn facilitates learning.…”

The impact of reward and punishment on skill learning depends on task demands

“…The majority of neuroscience research has focused on studying the effects of reward and punishment on decision-making123. However, in recent years interest in using reward and punishment to augment motor skill learning has surged456789 raising the enticing possibility that valenced feedback could be implemented in rehabilitation settings to improve physical therapy outcomes10111213. However, the variation in methodologies, performance metrics, and retention timescales used across different studies make establishing general principles challenging.…”

Basal ganglia contributions during the learning of a visuomotor rotation: Effect of dopamine, deep brain stimulation and reinforcement

“…Visuomotor adaptation is also believed to be mediated by the cerebellum and reinforcement based motor learning by the basal ganglia (Doya, ). In support of this notion, patients with hereditary or acquired cerebellar damage have selective impairment in supervised (error‐based) learning (Criscimagna‐Hemminger, Bastian, & Shadmehr, ; Donchin et al., ; Martin, Keating, Goodkin, Bastian, & Thach, ; Maschke, ; Smith, ; Tseng, Diedrichsen, Krakauer, Shadmehr, & Bastian, ) however, they show no impairment during reinforcement learning of the same task (Izawa, Criscimagna‐Hemminger, & Shadmehr, ; Therrien et al., ).…”

“…The mechanism underlying visuomotor adaptation is believed to minimize the differences between predicted and actual sensory feedback (Miall & Wolpert, ; Shadmehr & Krakauer, ; Shadmehr, Smith, & Krakauer, ). Reinforcement learning is considered to occur by selecting the motor commands that maximize reward or minimize punishment (Dam, Kording, & Wei, ; Therrien, Wolpert, & Bastian, ; Wu, Miyamoto, Gonzales Castro, Ölveczky, & Smith, ). Evidence for the independence of visuomotor adaptation and reinforcement learning is derived from experimental studies of visuomotor rotations that result in a recalibration of an internal forward model for error‐based learning during adaptation but not for reinforcement based learning (Izawa & Shadmehr, ).…”