Reinforcement-Based Processes Actively Regulate Motor Exploration Along Redundant Solution Manifolds

Abstract

From baby's babbling to a songbird practicing a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species, neural firing, gait patterns, and reaching behaviour. Past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here we test a fundamentally different idea—that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration. Across three human reaching experiments we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore is beneficial while acting in uncertain environments, during motor development, or when recovering from a neurological disorder to discover and learn new motor actions.

Reinforcement learning during locomotion

“…We calculated the difference in ΔLSL values between subsequent strides separately after successful steps and unsuccessful steps. The trial-to-trial change measure was calculated as the standard deviation of these changes, normalized by baseline trial-to-trial change variability (Cashaback et al, 2019; Roth et al, 2023), again to account for increases in this variability relative to baseline. We denote this measure as σ trial-to-trial .…”

“…While parsimonious, motor variability is not the only way to characterize exploration in movement-based reinforcement learning experiments. Some methods have characterized exploration as a random walk (Haith and Krakauer, 2014;Roth et al, 2023), but most methods have focused on the magnitude or variability of trial-to-trial of changes after successful vs unsuccessful movements (i.e., win-stay lose shift methods; Pekny et al, 2015;Therrien et al, 2016;Cashaback et al, 2019;Uehara et al, 2019;van Mastrigt et al, 2020van Mastrigt et al, , 2021. However, no studies have compared the latter measure of exploration while learning from reward prediction error versus correcting for target errors.…”

Reinforcement Learning during Locomotion

“…e N e u r o A c c e p t e d M a n u s c r i p t after successful steps and unsuccessful steps. The trial-to-trial change measure was calculated as the standard deviation of these changes, normalized by baseline trial-to-trial change variability (Cashaback et al, 2019;Roth et al, 2023), again to account for increases in this variability relative to baseline. We denote this measure as σtrial-to-trial.…”

“…While parsimonious, motor variability is not the only way to characterize exploration in movement-based reinforcement learning experiments. Some methods have characterized exploration as a random walk (Haith and Krakauer, 2014;Roth et al, 2023), but most methods have focused on the magnitude or variability of trial-to-trial of changes after successful vs unsuccessful movements (i.e., win-stay lose shift methods; Pekny et al, 2015;Therrien et al, 2016;Cashaback et al, 2019;Uehara et al, 2019;van Mastrigt et al, 2020van Mastrigt et al, , 2021. However, no studies have compared the latter measure of exploration while learning from reward…”

“…While we found that the RPE group had slightly greater trial-to-trial variability after unsuccessful steps compared to the TE group, the effect was relatively small compared to the effect in baseline-normalized motor variability. It is possible other models of exploration, for example, random walk behavior (Roth et al, 2023), may better distinguish between reward prediction error learning and target error correction. Or perhaps these methods of exploration could be task dependent.…”