Human Teleoperation - A Haptically Enabled Mixed Reality System for Teleultrasound

Abstract

<div> <div> <div> <p>Current teleguidance methods include verbal guidance and robotic teleoperation, which present tradeoffs between precision and latency versus flexibility and cost. We present a novel concept of "human teleoperation" which bridges the gap between these two methods. A prototype teleultrasound system was implemented which shows the concept’s efficacy. An expert remotely "teloperates" a person (the follower) wearing a mixed reality headset by controlling a virtual ultrasound probe projected into the person’s scene. The follower matches the pose and force of the virtual device with a real probe. The pose, force, video, ultrasound images, and 3-dimensional mesh of the scene are fed back to the expert. In this control framework, the input and the actuation are carried out by people, but with near robot-like latency and precision. This allows teleguidance that is more precise and fast than verbal guidance, yet more flexible and inexpensive than robotic teleoperation. The system was subjected to tests that show its effectiveness, including mean teleoperation latencies of 0.27 seconds and errors of 7 mm and 6◦ in pose tracking. The system was also tested with an expert ultrasonographer and four patients and was found to improve the precision and speed of two teleultrasound procedures. </p> </div> </div> </div>

Evaluation of Communication and Human Response Latency for (Human) Teleoperation

“…IV. DISCUSSION AND CONCLUSIONS In the original system described in [1], latency tests on the ROS WebSocket implementation showed on average 11.4 ms delay for pose and force transmission over a local network. In contrast, the new system has delays of 1.07 ± 0.57 ms over Ethernet or 5.80 ± 3.30 ms over WiFi on average for the same local network.…”

“…To the best of our knowledge, no other study has explored the human response when guided by an MR interface. To perform these tests, the prototype developed in [1] was improved to take direct force input from the expert, and a dummy ultrasound probe was designed for the follower, including 6 axis force/torque sensing and 6-DOF position and orientation (pose) tracking (Section II-B). A novel visual control system for the forces was developed, as described in Section II-C. Further rigorous tests of the visual force control and human tracking ability are presented in [31].…”

“…The effectiveness of the approach was demonstrated first using a WebSocket server and Robot Operating System (ROS) on a local wireless network (WLAN), showing large improvement in accuracy and completion time compared to existing teleguidance methods [1]. In the following sections, we describe and characterize a much improved communication system for human teleoperation, testing it in different networks, signal conditions, and data throughputs.…”

“…Finally, initial results in [1] showed that the system's latency was limited not by the communication latency, but rather by the reaction time of the follower. To quantify this carefully, a series of step response tests were carried out for force and position tracking of the follower.…”

“…Robotic teleoperation and video conference-based teleguidance fall on either end of a spectrum from performance to ease of use and deployment, leaving a large gap for solutions that are both flexible and easy to use and precise and efficient. In a previous paper [1], we introduced a novel concept of "Human Teleoperation" through mixed reality (MR) which bridges this gap. In this control framework, the human follower is controlled as a flexible, cognitive robot such that both the input and the actuation are carried out by people, but with near robot-like latency and precision.…”