A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision

Abstract

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-toend hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-toimage translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Novel Gesture-Based Robot Programming Approach with the Ability of Code Reuse

“…However, there are also quite non-standard solutions. For example, to solve occlusion problems, in [20], the authors used an active machine vision system to capture a human hand from the most optimal angle and distance. The active machine vision system consisted of the Universal Robotics UR5 collaborative robot, on which the Intel RealSense SR300 camera was installed.…”

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

“…Recent works have expanded the scope of teleoperating a single robot hand to complete arm-hand systems [16,25,64]. However, these systems are designed and engineered towards a particular robot model (e.g., Kuka arm with Allegro hand in [16], and PR2 arm with Shadow hand in [26]), and rely on retargeting or collision detection models trained for specific robot hardware (e.g., Allegro hand and XArm6) [16,52] , making them difficult to transfer to new arm-hand systems and new environments. In contrast, our system is highly modularized with a versatile hand-tracking solution compatible with an arbitrary number of cameras, and configurable robot hand retargeting and motion generation modules for easy adaption to various robot arms and robot hand choices.…”

“…In teleoperation, human operators often require a clear understanding of the spatial relationships between objects and robots to make informed decisions. This information can be provided through multi-view rendering, which is a widely used technique in previous teleoperation works [26,42,59]. Our web-based viewer offers multi-view support to the operator by simply opening multiple browser windows.…”

“…Compared to gripper-based manipulators, teleoper-ating dexterous hand-arm systems poses unprecedented challenges and often requires specialized apparatus that comes with high costs and setup efforts, such as Virtual Reality (VR) devices [4,17,15], wearable gloves [29,30], handheld controller [45,46,20], haptic sensors [12,23,50,53], or motion capture trackers [65]. Fortunately, recent developments in vision-based teleoperation [2,24,16,26,42,27,21,22,3] have provided a low-cost and more generalizable alternative for teleoperating dexterous robot systems.…”