Clustering analysis of movement kinematics in reinforcement learning

Abstract

Reinforcement learning has been used as an experimental model of motor skill acquisition, where at times movements are successful and thus reinforced. One fundamental problem is to understand how humans select exploration over exploitation during learning. The decision could be influenced by factors such as task demands and reward availability. In this study, we applied a clustering algorithm to examine how a change in the accuracy requirements of a task affected the choice of exploration over exploitation. Participants made reaching movements to an unseen target using a planar robot arm and received reward after each successful movement. For one group of participants, the width of the hidden target decreased after every other training block. For a second group, it remained constant. The clustering algorithm was applied to the kinematic data to characterize motor learning on a trial-to-trial basis as a sequence of movements, each belonging to one of the identified clusters. By the end of learning, movement trajectories across all participants converged primarily to a single cluster with the greatest number of successful trials. Within this analysis framework, we defined exploration and exploitation as types of behaviour in which two successive trajectories belong to different or similar clusters, respectively. The frequency of each mode of behaviour was evaluated over the course learning. It was found that by reducing the target width, participants used a greater variety of different clusters and displayed more exploration than exploitation. Excessive exploration relative to exploitation was found to be detrimental to subsequent motor learning.

Narrowing the coordination solution space during motor learning standardizes individual patterns of search strategy but diversifies learning rates

“…Those explored coordination patterns are strongly anchored for a period of time (i.e., with the highest probability to transit towards them), but still are not the final destination of the learner, which fits the definition of a local strategy 1 . This exploration dynamics, as an information gathering process is more than merely a global trial to trial variability of the behavior during practice as proposed previously 19 , 30 . Indeed, a global ratio between the transition of coordination patterns between trials and the repetition of similar coordination patterns between trial (e.g., the exploration/exploitation ratio 19 , 30 ) does not account for this exploratory dynamics during learning as it does not account for the order or sequencing of the transitions.…”

Learning the arm-leg coordination in breaststroke swimming: Identifying exploration of movement dynamics with drifting Markov Models

“…In fact, even if some patterns are only little explored, it seems bene cial for the learner to still be able to explore them as they may rapidly gather information from their exploration. The initial exibility of a learner could therefore form the basis for an effective exploration during learning, as recently highlighted by Sidarta et al 25 . Narrowing the region of exploration by limiting the potential patterns that can be visited during learning can impact the individual dynamics of the performance because the individually necessary information therefore cannot be gathered through an effective exploration 38 .…”

“…Those explored coordination patterns are strongly anchored for a period of time (i.e., with the highest probability to transit towards them), but still are not the nal destination of the learner, which ts the de nition of a local strategy 1 . This exploration dynamics, as an information gathering process is more than merely a global trial to trial variability of the behavior during practice as proposed previously 19,25 . Indeed, a global ratio between the transition of coordination patterns between trials and the repetition of similar coordination patterns between trial (e.g., the exploration/exploitation ratio 19,25 ) does not account for this exploratory dynamics during learning as it does not account for the order or sequencing of the transitions.…”

“…From the clustering, each trial was labelled with a speci c exhibited coordination pro le. The time series of those exhibited coordination were re-constructed putting one cycle after the previous one in the chronological order they were performed, representing the successive behaviours that were exhibited by a learner 18,25 . Those time series of labelled behaviours were thereafter modelled using Drifting Markov Models (DMM) 51 .…”

Evaluating transfer prediction using machine learning for skill acquisition study under various practice conditions

“…This result suggests that more structured variations could more effectively guide learning. Moreover, recent results demonstrated that practice conditions leading to excessive exploration of movement solutions can be detrimental to learning (Sidarta et al, 2022). In fact, newly discovered behavioral solutions require exploitation during practice to stabilize them in the learners' repertoire (Hossner et al, 2016;Komar et al, 2019).…”