The Effects of Predictive Displays on Performance in Driving Tasks with Multi-Second Latency

Abstract

Tele-operation of a Lunar rover from a control station on Earth involves a latency of several seconds due primarily to the finite speed (light-speed) of command and sensor signals, and this latency creates a difficult control task for the human operator. Two predictive displays, which seek to aid viewer perception of present events, were designed and evaluated for the specific task of driving a rover with multi-second latency. These displays provided visual information to the human operator on the rover's real-time locomotion, as predicted from control inputs executed by the operator. A human-subject experiment with 12 participants was conducted in which the participants navigated an actual rover through obstacle courses. There were four experimental conditions repeated by each participant: (1) delayed video feed only, (2, 3) two predictive displays based on delayed video feed, and (4) a reference condition of video feed with no delay. Inferential statistics show that both predictive displays significantly improved performance in terms of time taken to complete the courses, and one of the displays facilitated performance approaching that with no delay. No trends were observed in terms of collisions with or encroachments near obstacles.

A shared control method for obstacle avoidance with mobile robots and its interaction with communication delay

“…For example, augmented reality has been used to assist the human operator in estimating the robot’s actual location. Studies have shown that predictive displays have been helpful for driving tasks with both short delays (approximately 400–1100 ms) (Davis et al, 2010) and multi-second delays (Matheson et al, 2013). Augmented reality has also been used to render 3D video projections and create simulated exocentric views of the robot’s environment that improve teleoperation performance (Ferland et al, 2009; Kelly et al, 2011).…”

Long future frame prediction using optical flow‐informed deep neural networks for enhancement of robotic teleoperation in high latency environments

“…Considering the impacts of latency on teleoperation performance, a significant research effort has been made to the design of predictive interface‐based teleoperation systems (Deng & Jagersand, 2003; Dybvik et al, 2020; Ha et al, 2018; Matheson et al, 2013; Ricks et al, 2004; C. Wang et al, 2016; Wilde et al, 2020; Witus et al, 2011; Y. Xiong et al, 2006). These predictive interfaces have attempted to compensate latency through first‐order predictive interfaces that mostly offer the predicted state variables for the period of delay (Artstein, 1982; Manitius & Olbrot, 1979).…”

Predictive display for teleoperation with virtual reality fusion technology

“…To achieve consistency between the virtual world and the real world, augmented reality (AR) technology is used, and error calibration technology is used to approximate model matching [15]. Matheson et al [16] verified the performance of the predicted display by driving the rover through an obstacle course with a delay of 3 s. Xia et al [17] introduced virtual fixtures into the model-based remote control system. Zhu et al [18] employed a Kinect camera to create a virtual target object.…”