Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Abstract

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-inhand systems automatically control the hand aperture and preshaping before the grasp, based on visual input coming from a camera on the wrist. In this work, we present an eye-inhand learning-based approach for hand pre-shape classification from RGB sequences. In order to reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories for the purpose. We tackle the peculiarity of the eye-in-hand setting by means of a model for the human arm trajectories, with domain randomization over relevant visual elements. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. Our code, real and synthetic datasets will be released upon acceptance.

Affordance Segmentation Using Tiny Networks for Sensing Systems in Wearable Robotic Devices

“…Likewise, [20] augmented affordance segmentation with keypoint detection. A simulator included in [21] generated a collection of grasping sequences, while [13] involved a large dataset with RGBD information. Synthetic images were also considered in [22].…”

Hand Grasp Pose Prediction Based on Motion Prior Field

“…Many studies have involved attachment of a camera to the prosthetic hand to capture objects and environmental information to assist the user with pre-shape adjustment of the prosthetic hand or control of the wrist joint. The authors of [ 2 , 3 ] installed a depth camera on the wrist to capture object information to realize the adjustment of the wrist angle and the selection of pre-grasp gestures. Other studies, such as [ 4 , 5 , 6 , 7 ], involved installation of RGB cameras on prosthetic hands, relying only on RGB images to achieve the same results.…”

“…First, current research does not consider the prediction of grasp pose. In [ 3 , 6 ], for example, the essence of the work is consideration of the object–gesture classification problem, which requires the user to set up the appropriate approach posture during the pre-shape process. The aim is only to grasp, but there is no consideration of whether the different contact areas and different approach directions are compatible with everyday human habits.…”

“…Additionally, the pre-shape performance mainly depends on the camera capture perspective. In [ 3 , 4 ], a predictive model is used to learn directly from the captured image, making the application highly viewpoint-dependent and, thus, difficult to generalize to multi-view deployments.…”