High Latency Unmanned Ground Vehicle Teleoperation Enhancement by Presentation of Estimated Future through Video Transformation

Abstract

Long-distance, high latency teleoperation tasks are difficult, highly stressful for teleoperators, and prone to over-corrections, which can lead to loss of control. At higher latencies, or when teleoperating at higher vehicle speed, the situation becomes progressively worse. To explore potential solutions, this research work investigates two 2D visual feedback-based assistive interfaces (sliding-only and sliding-and-zooming windows) that apply simple but effective video transformations to enhance teleoperation. A teleoperation simulator that can replicate teleoperation scenarios affected by high and adjustable latency has been developed to explore the effectiveness of the proposed assistive interfaces. Three image comparison metrics have been used to fine-tune and optimise the proposed interfaces. An operator survey was conducted to evaluate and compare performance with and without the assistance. The survey has shown that a 900ms latency increases task completion time by up to 205% for an on-road and 147% for an off-road driving track. Further, the overcorrection-induced oscillations increase by up to 718% with this level of latency. The survey has shown the sliding-only video transformation reduces the task completion time by up to 25.53%, and the sliding-and-zooming transformation reduces the task completion time by up to 21.82%. The sliding-only interface reduces the oscillation count by up to 66.28%, and the sliding-and-zooming interface reduces it by up to 75.58%. The qualitative feedback from the participants also shows that both types of assistive interfaces offer better visual situational awareness, comfort, and controllability, and significantly reduce the impact of latency and intermittency on the teleoperation task.

Long future frame prediction using optical flow‐informed deep neural networks for enhancement of robotic teleoperation in high latency environments

“…Therefore, we have designed a system that can simulate high latency teleoperation. Our previous paper (Moniruzzaman et al, 2022) proposes a conventional video transformation technique to aid teleoperators in high latency teleoperation conditions. In this manuscript we aim to show that neural network‐predicted and generated future frames can achieve similar or even better results considering the pixel and structural integrity‐based comparison of predicted future frames with the ground truth.…”

“…Lag or latency as low as 10–20 ms can be noticeable by teleoperators, and more than 225 ms has been shown to increase operator action time by 64% and error rate by 214% (MacKenzie & Ware, 1993). Our recent study (Moniruzzaman et al, 2022) demonstrates that ground vehicle teleoperation at speeds above 10 kph becomes almost impossible if the latency exceeds 1 s. For teleoperating ground vehicles under these conditions, latencies of as low as 170 ms can significantly degrade the vehicle control, primarily due to the effects of oversteering and subsequent delayed compensation (Frank et al, 1988).…”

“…This model is capable of creating and saving video data sets of ground vehicle teleoperation along with the control input time series for AI and deep learning‐based model training. The model developed for and used in this research is an extension of the model discussed in our previous work (Moniruzzaman et al, 2022). In a novel approach to long future prediction through a generative adversarial network (GAN)‐based deep neural network, we have used image‐to‐image translation techniques along with optical flow as supporting information for the network training and inference.…”

“…Optical flow generation is used to reflect the control input time‐series signals. In our model, this is generated by integrating an Unreal Engine (UE) driving simulation to our previously proposed teleoperation simulator (Moniruzzaman et al, 2022). For long future prediction, we have used the Pix2Pix deep network discussed in Isola et al (2017).…”

“…Conventional predictive teleoperation enhancement has been investigated for both 2D and 3D predictive displays. Different implementations for both of these predictive display‐based approaches are summarized in Table 1 (adapted from Moniruzzaman et al, 2022) and discussed below.…”