Methods for UGV teleoperation with high latency communications

Abstract

In this project, we developed and demonstrated complementary UGV control methods for teleoperation with highlatency communications. The methods included latency protection, predictive displays, and supervisory control. Latency protection mitigate against typical types of high-latency teleoperation input errors. The Phase I latency protection methods included filtering the joysick commands, limiting the commanded rates as a function of latency, and emergency stop when the operator commands and OCU navigation video were out of phase. Predictive displays indicate to the operator the current state of the UGV, i.e., the state after all of the latent commands are executed (latent commands are those that have been issued but whose effects do not yet appear in the OCU display). We implemented two alternative predictive display methods: augmented reality using iconography to indicate the effects of the latent commands, and virtual reality which warps the image to show the view to reflect the latent commands. Supervisory control allows the operator to specify simple, short-range objectives that the UGV can accomplish on its own, without advanced sensing, path planning, etc. We implemented an effective "point-and-go" supervisory control system. We successfully implemented and demonstrated these methods on a Packbot© 510 EOD robot (made by iRobot Corporation), currently being used in-theater.

Long future frame prediction using optical flow‐informed deep neural networks for enhancement of robotic teleoperation in high latency environments

“…The concept of the predictive display has helped earlier approaches such as , T A B L E 1 Summary of predictive feedback-based teleoperation enhancement techniques (adapted from Moniruzzaman et al, 2022) Author and year Technique Robot type Two-dimensional predictive feedback | 395 , Buzan andSheridan (1989), andHirzinger et al (1989) to develop teleoperation systems that provide the operator information about the response of the system before it actually happens and hence avoids possible collisions. Iconography has also been used as a means to implement robotic state prediction for more recent teleoperation user interfaces, such as Augmented Reality (AR) and Virtual Reality (VR; Witus et al, 2011). However, these early approaches do not provide an intuitive control interface as they were mostly predictions based on non-video-based feedback and are not suitable for higher-speed operation.…”

“…Considering the impacts of latency on teleoperation performance, a significant research effort has been made to the design of predictive interface‐based teleoperation systems (Deng & Jagersand, 2003; Dybvik et al, 2020; Ha et al, 2018; Matheson et al, 2013; Ricks et al, 2004; C. Wang et al, 2016; Wilde et al, 2020; Witus et al, 2011; Y. Xiong et al, 2006). These predictive interfaces have attempted to compensate latency through first‐order predictive interfaces that mostly offer the predicted state variables for the period of delay (Artstein, 1982; Manitius & Olbrot, 1979).…”

“…Considering the impacts of latency on teleoperation performance, a significant research effort has been made to the design of predictive interface-based teleoperation systems (Deng & Jagersand, 2003;Dybvik et al, 2020;Ha et al, 2018;Matheson et al, 2013;Ricks et al, 2004;C. Wang et al, 2016;Wilde et al, 2020;Witus et al, 2011;Y. Xiong et al, 2006).…”

Guidance Point Generation-Based Cooperative UGV Teleoperation in Unstructured Environment

“…Under the guidance interactive cooperative control mode, there is a hierarchical cooperative relationship between the operator and the navigation control system [24]. The operator makes a control decision according to the statues and environmental perception information from the UGV, and input corresponding global or local guidance information to navigate the UGV maneuvering toward target position.…”