A Neural Dynamic Architecture for Reaching and Grasping Integrates Perception and Movement Generation and Enables On-Line Updating

Abstract

Reaching for objects and grasping them is a fundamental skill for any autonomous robot that interacts with its environment. Although this skill seems trivial to adults, who effortlessly pick up even objects they have never seen before, it is hard for other animals, for human infants, and for most autonomous robots. Any time during movement preparation and execution, human reaching movement are updated if the visual scene changes (with a delay of about 100 ms). The capability for online updating highlights how tightly perception, movement planning, and movement generation are integrated in humans. Here, we report on an effort to reproduce this tight integration in a neural dynamic process model of reaching and grasping that covers the complete path from visual perception to movement generation within a unified modeling framework, Dynamic Field Theory. All requisite processes are realized as time-continuous dynamical systems that model the evolution in time of neural population activation. Population level neural processes bring about the attentional selection of objects, the estimation of object shape and pose, and the mapping of pose parameters to suitable movement parameters. Once a target object has been selected, its pose parameters couple into the neural dynamics of movement generation so that changes of pose are propagated through the architecture to update the performed movement online. Implementing the neural architecture on an anthropomorphic robot arm equipped with a Kinect sensor, we evaluate the model by grasping wooden objects. Their size, shape, and pose are estimated from a neural model of scene perception that is based on feature fields. The sequential organization of a reach and grasp act emerges from a sequence of dynamic instabilities within a neural dynamics of behavioral organization, that effectively switches the neural controllers from one phase of the action to the next. Trajectory formation itself is driven by a dynamical systems version of the potential field approach. We highlight the emergent capacity for online updating by showing that a shift or rotation of the object during the reaching phase leads to the online adaptation of the movement plan and successful completion of the grasp.

Integrating Brain and Biomechanical Models—A New Paradigm for Understanding Neuro-muscular Control

“…Nevertheless, the biomechanically accurate motor system they describe has the potential to form part of a computational neurobehavioral study of swimming behavior in the fish, if it were combined in a loop with suitable sensory input and sensory processing models. Knips et al ( 2017 ) is a report of a reach-and-grasp robot arm controlled via a dynamic neural field brain model. The sensory input for this system—its “eyes”—is a Microsoft Kinect sensor; it also has somatosensory feedback from the fingers of the robot's hand.…”

“…While this robot has closed-loop control and is clearly inspired by biology, it remains a study of robotics and of the improvement of the control of the robot's reach-and-grasp function, rather than a study which aims to learn more about the biology of a primate arm. For this reason, we would describe the study of Knips et al ( 2017 ) as an embodied model.…”

What the Dynamic Systems Approach Can Offer for Understanding Development: An Example of Mid-childhood Reaching

“…To understand developmental changes in reaching, the attractor landscape of reaching has to be identified (e.g., Schöner, 1990 , 1994 ; Huys et al, 2014 ; Knips et al, 2017 ). Following Schöner (1990 , 1994 ), we suggest that discrete reaching movements can be conceptualized as sequentially stabilizing point attractors (i.e., representing the initial and target location) and limit-cycle attractors (i.e., representing the movement) within a single dynamic system.…”

Autonomous Sequence Generation for a Neural Dynamic Robot: Scene Perception, Serial Order, and Object-Oriented Movement

“…This requires that an object's visual coordinates are transformed into coordinates anchored in the initial position of the hand (Schöner et al, 2019). We show how such a neural representation of movement targets may be linked to the visual array, enabling online updating of movement generation when the scene changes (see Knips et al, 2017 for an earlier version of such online updating).…”