Long future frame prediction using optical flow‐informed deep neural networks for enhancement of robotic teleoperation in high latency environments

Abstract

High latency in teleoperation has a significant negative impact on operator performance. While deep learning has revolutionized many domains recently, it has not previously been applied to teleoperation enhancement. We propose a novel approach to predict video frames deep into the future using neural networks informed by synthetically generated optical flow information. This can be employed in teleoperated robotic systems that rely on video feeds for operator situational awareness. We have used the image-to-image translation technique as a basis for the prediction of future frames. The Pix2Pix conditional generative adversarial network (cGAN) has been selected as a base network. Optical flow components reflecting real-time control inputs are added to the standard RGB channels of the input image. We have experimented with three data sets of 20,000 input images each that were generated using our custom-designed teleoperation simulator with a 500-ms delay added between the input and target frames. Structural Similarity Index Measures (SSIMs) of 0.60 and Multi-SSIMs of 0.68 were achieved when training the cGAN with three-channel RGB image data. With the five-channel input data (incorporating optical flow) these values improved to 0.67 and 0.74, respectively. Applying Fleiss' κ gave a score of 0.40 for three-channel RGB data, and 0.55 for fivechannel optical flow-added data. We are confident the predicted synthetic frames are of sufficient quality and reliability to be presented to teleoperators as a video feed that will enhance teleoperation. To the best of our knowledge, we are the first to attempt to reduce the impacts of latency through future frame prediction using deep neural networks.

Structure‐Aware Image Translation‐Based Long Future Prediction for Enhancement of Ground Robotic Vehicle Teleoperation

“…This indicates that with reference to the image quality for all of our improvements to the future prediction technique described in ref. [17], the predicted frames generated by our different models follow a normal and almost symmetrical distribution. We can see that the standard deviations for all evaluation metrics and combinations are very low.…”

“…A detailed description of the simulator and the human teleoperator involved data collection process is described in ref. [17]. Similar to the previously collected data, we have used the OBS software for the synchronized video data recording process.…”

“…Therefore, similar to ref. [17], we have extracted optical flow for the delayed frames of both the training and testing tests. We have extracted the dense or per-pixel optical flow from two consecutive frames through the Gunner-Farneback [47] method.…”

“…For predicting and generating synthetic future frames, the generator loss plays a more significant role and modifications to the generator loss function can have a direct impact on the quality of the synthetic image. For our previous article, [17] the generator loss function was…”

“…Figure 6. Comparisons of the delayed, predicted, and ground truth frames for different trained models: a) the model presented in ref [17]. trained with Forza_GT þ UE þ Std_DL data, b) our modified network with SSIM perpetual þ L1 loss (50% weightage each), c) modified network with SSIM perpetual loss function, and d) modified network with L1 loss function.…”