Interactions between motor exploration and reinforcement learning

Abstract

Motor exploration, a trial-and-error process in search for better motor outcomes, is known to serve a critical role in motor learning. This is particularly relevant during reinforcement learning, where actions leading to a successful outcome are reinforced while unsuccessful actions are avoided. Although early on motor exploration is beneficial to finding the correct solution, maintaining high levels of exploration later in the learning process might be deleterious. Whether and how the level of exploration changes over the course of reinforcement learning, however, remains poorly understood. Here we evaluated temporal changes in motor exploration while healthy participants learned a reinforcement-based motor task. We defined exploration as the magnitude of trial-to-trial change in movements as a function of whether the preceding trial resulted in success or failure. Participants were required to find the optimal finger-pointing direction using binary feedback of success or failure. We found that the magnitude of exploration gradually increased over time when participants were learning the task. Conversely, exploration remained low in participants who were unable to correctly adjust their pointing direction. Interestingly, exploration remained elevated when participants underwent a second training session, which was associated with faster relearning. These results indicate that the motor system may flexibly upregulate the extent of exploration during reinforcement learning as if acquiring a specific strategy to facilitate subsequent learning. Also, our findings showed that exploration affects reinforcement learning and vice versa, indicating an interactive relationship between them. Reinforcement-based tasks could be used as primers to increase exploratory behavior leading to more efficient subsequent learning. NEW & NOTEWORTHY Motor exploration, the ability to search for the correct actions, is critical to learning motor skills. Despite this, whether and how the level of exploration changes over the course of training remains poorly understood. We showed that exploration increased and remained high throughout training of a reinforcement-based motor task. Interestingly, elevated exploration persisted and facilitated subsequent learning. These results suggest that the motor system upregulates exploration as if learning a strategy to facilitate subsequent learning.

Quantifying exploration in reward-based motor learning

“…Using a weight-shifting task, we replicated a finding from studies using arm movements: increased variability following non-rewarded trials as compared to rewarded trials [4,5,[8][9][10][11][12][13]. In addition, we found a ratio between variability due to exploration and sensorimotor noise of 0.78.…”

“…Therefore, this finding implies that studying sensorimotor noise and exploration requires a high number of trials. In the literature, human motor learning experiments usually seem to contain up to 200 feedback trials [8,10,12,[31][32][33][34][35][36], although some studies report higher numbers, up to 900 trials [4,5,11,16,19,37,38]. An example of a study incorporating many trials is [14], in which rats performed 300k trials on average.…”

“…If you want to improve your performance but receive no feedback on error size and direction, you have to explore which actions lead to reward such that rewarded actions can be exploited [1,7]. Indeed, studies into reward-based motor learning have shown a higher variability in motor output following non-rewarded movements than following rewarded movements [4,5,[8][9][10][11][12][13][14]. Moreover, there are indications that initial variability in motor output, regarded as exploration, is positively related with learning, both in songbirds [15] and in humans [16].…”

“…To eliminate the influence of learning on variability, feedback was stochastic. To verify whether this task results in exploration, we first aimed to replicate the finding that variability following non-rewarded trials is higher than variability following rewarded trials [4,5,[8][9][10][11][12][13][14]. Next, we aimed to compare the estimation of sensorimotor noise in three feedback contexts: (1) trials in a baseline phase without feedback, (2) trials in short blocks without feedback that were flanked by short blocks of feedback trials and (3) rewarded trials in blocks of feedback trials.…”

Cell-type specific outcome representation in primary motor cortex

“…The outcome signals we report here, may be considered as higher level signals reporting the end result which may have relevance in correcting future movements but have no relevance to immediate movements. This type of reinforcement outcome signals can serve for motor learning and adaptation assigning credit or punishment in the process of learning over longer time scales as discussed below (Even- Chen et al, 2017;Schultz, 2000;Wolpert et al, 2011) and for maintaining learned motor skills (Uehara et al, 2019). In principle, this success and failure related neuronal activity might result from kinematic differences between success and failure trials.…”