Credit assignment in movement-dependent reinforcement learning

Abstract

When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tractable solution to the credit assignment problem.decision-making | reinforcement learning | sensory prediction error | reward prediction error | cerebellum W hen a diner reaches across the table and knocks over her coffee, the absence of anticipated reward should be attributed to a failure of coordination rather than diminish her love of coffee. Although this attribution is intuitive, current models of decision-making lack a mechanistic explanation for this seemingly simple computation. We set out to ask if, and how, selection processes in decision-making incorporate information specific to action execution and thus solve the credit assignment problem that arises when an expected reward is not obtained because of a failure in motor execution.Humans are highly capable of tracking the value of stimuli, varying their behavior on the basis of reinforcement history (1, 2), and exhibiting sensitivity to intrinsic motor noise when reward outcomes depend on movement accuracy (3-5). In real-world behavior, the underlying cause of unrewarded events is often ambiguous: A lost point in tennis could occur because the player made a poor choice about where to hit the ball or failed to properly execute the stroke. However, in laboratory studies of reinforcement learning, the underlying cause of unrewarded events is typically unambiguous, either solely dependent on properties of the stimulus or on motor noise. Thus, it remains unclear how people assign credit to either extrins...

Distinct Neural Signatures of Outcome Monitoring following Selection and Execution Errors

“…Overall, this bandit was chosen on 39.49% (±9.38) of the trials, which was significantly greater than the Low Execution/High Selection error bandit (29.01%; ±6.89%, p < .001) and Neutral bandit (31.50%; ±8.64%, p = .046), with no difference for the latter two (p = .877). Consistent with previous work, when expected value is equal, participants prefer choices in which unrewarded trials are attributed to errors in movement execution rather than errors in action selection (Wu et al, 2009;Green et al, 2010;McDougle et al, 2016;Parvin et al, 2018;McDougle et al, 2019).…”

“…Behaviorally, participants showed lower switch rates following execution errors, a pattern consistent with the hypothesis that the reinforcement learning system discounts these errors (McDougle et al, 2016;. Contrary to our prediction that the MFN would be attenuated following execution errors, these errors produced the largest MFNs.…”

“…Although the majority of neuroscience decision-making research has focused almost exclusively on the former, a number of studies have shown that failed outcomes attributed to sensorimotor errors can markedly alter choice preferences (Green et al, 2010;McDougle et al, 2016;Parvin et al, 2018;McDougle et al, 2019). It has been postulated that these shifts arise from error signals credited to the sensorimotor system attenuating the operation of reinforcement learning processes (McDougle et al, 2016). Here, we examined whether a putative ERP signature of reinforcement learning, the medial frontal negativity (MFN), varied in response to selection and motor errors.…”

Enhanced visuomotor learning and generalization in expert surgeons

“…Another, more probable explanation is that minimally invasive surgeons are exploiting additional resources such as explicit strategies or heuristics to adapt faster and more completely than controls. Although a long-standing view has been that visuomotor adaptation reflects incremental (i.e., trial-by-trial) error-based learning that occurs in an implicit and automatic manner, recent experimental and computational modelling work indicates that it often depends on the operation of multiple interacting learning processes [32,33]. For example, if a perturbation induces a large reaching error to the right side of the target, subjects experiencing this error signal might consciously choose to aim to the left side of the target on the next trial in order to compensate.…”

Optimizing motor decision-making through competition with opponents

“…For example, selecting an appropriate trajectory for reaching a glass of water can lead to a low risk of spilling water, and likewise, finding a running course to easily pass through in rugby and deciding the best aiming location in a tennis match can increase the possibility of victory in a competition. Despite the importance of optimal decision-making, for over a decade, sub-optimal and overly risk-seeking behaviours have been reported in various motor decision tasks [1][2][3][4][5][6][7][8][9] . Determining how to improve sub-optimal and risk-seeking decision-making behaviour is crucial to enhance well-being in daily life and performance in sports.…”