Robot-assisted motor training: Assistance decreases exploration during reinforcement learning

Abstract

Reinforcement learning (RL) is a form of motor learning that robotic therapy devices could potentially manipulate to promote neurorehabilitation. We developed a system that requires trainees to use RL to learn a predefined target movement. The system provides higher rewards for movements that are more similar to the target movement. We also developed a novel algorithm that rewards trainees of different abilities with comparable reward sizes. This algorithm measures a trainee's performance relative to their best performance, rather than relative to an absolute target performance, to determine reward. We hypothesized this algorithm would permit subjects who cannot normally achieve high reward levels to do so while still learning. In an experiment with 21 unimpaired human subjects, we found that all subjects quickly learned to make a first target movement with and without the reward equalization. However, artificially increasing reward decreased the subjects' tendency to engage in exploration and therefore slowed learning, particularly when we changed the target movement. An anti-slacking watchdog algorithm further slowed learning. These results suggest that robotic algorithms that assist trainees in achieving rewards or in preventing slacking might, over time, discourage the exploration needed for reinforcement learning.

Self-powered robots to reduce motor slacking during upper-extremity rehabilitation: a proof of concept study

“…Further, assist-as-needed control algorithms that require patient’s neurophysiological signals (e.g., EMG or EEG) to infer their intention to perform a particular movement (Brauchle et al, 2015; Rosen et al, 2001; Stein, 2009) are sensitive to electrode placement and skin properties, and often lack fidelity due to crosstalk from neighboring muscles or brain regions (Marchal-Crespo and Reinkensmeyer, 2009). More importantly, researchers have shown that incorporating sophisticated control algorithms (e.g., anti-slacking watchdog algorithms) could actually discourage motor exploration and learning over time (Sans-Muntadas et al, 2014). However, appropriately designed body-powered robots could potentially address many of these issues at a relatively low cost without compromising on safety.…”

Spatial diversity of spontaneous activity in the cortex

“…Neuromodulators like acetylcholine therefore perhaps affect plasticity not only by gating it (Gu, 2002 ; Chubykin et al, 2013 ; Chun et al, 2013 ), but also by modulating spontaneous activity and response variability (Zinke et al, 2006 ; Goard and Dan, 2009 ; Zhou et al, 2011 ). There is evidence that variability is needed for or enhances some forms of motor learning in people (Sans-Muntadas et al, 2014 ; Taylor and Ivry, 2014 ) and song birds (Woolley and Kao, 2015 ). Consequently, more work seems warranted to understand whether the variability associated with cortical spontaneous activity has a role in cortical plasticity, so that it is itself a factor shaping the functional diversity of the cortex.…”