Sensory Manipulation as a Countermeasure to Robot Teleoperation Delays: System and Evidence

Abstract

In the realm of robotics and automation, robot teleoperation, which facilitates human-machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.

Neural dynamics of delayed feedback in robot teleoperation: insights from fNIRS analysis

“…In our previous study Du et al ( 2023 ), we investigated various performance metrics to determine the influence of delayed feedback in teleoperation. The placement error, time on task, and cognitive load during pick-up and drop-off phases were evaluated using pupil size as a physiological indicator.…”

“…Building upon our detailed system design presented in Du et al ( 2023 ), this section offers a concise overview of the key components of our teleoperation system, focusing on the VR system, its integration with various elements, and the implementation of delay coding functions.…”

“…To prepare for more extreme conditions of delayed teleoperation, we have proposed an innovative approach to sensory manipulation. By utilizing a physics engine, we simulate synthetic force feedback in anticipation of the actual haptic signal data (Du et al, 2023 ). This method creates a more intuitive and responsive teleoperation experience, even when communication delays change.…”