Mixed Reality Human Teleoperation

Abstract

Recent work introduced the concept of human teleoperation (HT), where the remote robot typically considered in conventional bilateral teleoperation is replaced by a novice person wearing a mixed reality head mounted display and tracking the motion of a virtual tool controlled by an expert. HT has advantages in cost, complexity, and patient acceptance for telemedicine in low-resource communities or remote locations. However, the stability, transparency, and performance of bilateral HT are unexplored. In this paper, we therefore develop a mathematical model and simulation of the HT system using test data. We then analyze various control architectures with this model and implement them with the HT system to find the achievable performance, investigate stability, and determine the most promising teleoperation scheme in the presence of time delays. We show that instability in HT, while not destructive or dangerous, makes the system impossible to use. However, stable and transparent teleoperation are possible with small time delays (< 200 ms) through 3-channel teleoperation, or with large time delays through model-mediated teleoperation with local pose and force feedback for the novice.

Robust Object Pose Tracking for Augmented Reality Guidance and Teleoperation

“…This includes MR guidance of device assembly [1], maintenance and inspection [2], [3], physiotherapy and rehabilitation [4]- [6], needle biopsies and interventions [7], [8], surgical guidance [9], and even sports instruction [10], [11]. Direct "human teleoperation" of people through a tightly coupled hand-over-hand guidance system is also being developed [12], [13]. By fusing virtual content with the real environment, MR enables seamless instruction and guidance while completing tasks, and allows users to "see inside" patients by overlaying medical images.…”

Low-Profile 6-Axis Differential Magnetic Force/Torque Sensing

“…For teleoperation and remote guidance, haptic feedback is essential to make the interaction intuitive and realistic for the operator. Force/torque measurement is therefore particularly important in tele-robotic US procedures [1]- [5], or for "teleoperating" novice people in performing tests through a mixed reality interface [6], [7]. In addition, sonographers often face increased incidence of musculoskeletal injury due to the awkward application of loads during exams [8].…”