A novel model of motor learning capable of developing an optimal movement control law online from scratch

Abstract

A computational model of a learning system (LS) is described that acquires knowledge and skill necessary for optimal control of a multisegmental limb dynamics (controlled object or CO), starting from "knowing" only the dimensionality of the object's state space. It is based on an optimal control problem setup different from that of reinforcement learning. The LS solves the optimal control problem online while practicing the manipulation of CO. The system's functional architecture comprises several adaptive components, each of which incorporates a number of mapping functions approximated based on artificial neural nets. Besides the internal model of the CO's dynamics and adaptive controller that computes the control law, the LS includes a new type of internal model, the minimal cost (IM(mc)) of moving the controlled object between a pair of states. That internal model appears critical for the LS's capacity to develop an optimal movement trajectory. The IM(mc) interacts with the adaptive controller in a cooperative manner. The controller provides an initial approximation of an optimal control action, which is further optimized in real time based on the IM(mc). The IM(mc) in turn provides information for updating the controller. The LS's performance was tested on the task of center-out reaching to eight randomly selected targets with a 2DOF limb model. The LS reached an optimal level of performance in a few tens of trials. It also quickly adapted to movement perturbations produced by two different types of external force field. The results suggest that the proposed design of a self-optimized control system can serve as a basis for the modeling of motor learning that includes the formation and adaptive modification of the plan of a goal-directed movement.

Perceptual suboptimality: Bug or feature?

“…These steps are an essential component of the behavioral paradigm, and therefore, the cost of the corresponding neural effort needs to be included in the optimality criterion. In the field of movement control, where an optimality approach has successfully accounted for vast experimental data (for reviews, see Shimansky et al 2004;Todorov 2004), the consideration of the cost of information processing was critical for understanding even relatively simple motor behaviors such as reaching to grasp and point-to-point movements . Also, the "trailing" pattern of joint coordination typically observed during well-learned arm movements can be fully understood only if neural effort for joint coordination is considered as a primary component of the optimality criterion (Dounskaia & Shimansky 2016;Goble et al 2007).…”

From task parameters to motor synergies: A hierarchical framework for approximately optimal control of redundant manipulators

“…Desired trajectory planning implies that the neural processing in the mosaic of brain areas involved in online sensorimotor control does little more than play a prerecorded movement tape -which is unlikely [4]. Consequently, we and others [11,[22][23][24][25] have focused on optimal feedback control models, that predict not only average behavior but also the taskspecific sensorimotor contingencies used to generate intelligent adjustments online. Such adjustments enable biological systems to "solve a control problem repeatedly rather than repeat its solution" [26], and thus afford remarkable levels of performance in the presence of noise, delays, internal fluctuations, and unpredictable changes in the environment.…”

Altered tuning in primary motor cortex does not account for behavioral adaptation during force field learning

“…In either case, control signals must ultimately be expressed in the intrinsic, musculoskeletal coordinates of motor execution. Both approaches posit the use of internal models of the limb (Wolpert et al 1995), whether to transform a desired kinematic trajectory into kinetic commands (Kalaska and Crammond 1992; Shadmehr and Mussa-Ivaldi 1994), or for optimal state estimation (Shimansky et al 2004; Todorov 2004; Scott 2012). These internal models are learned during development, but it is also possible to modify them on a shorter time scale, for example, when using a heavy tool that requires altered forces to produce a given movement.…”