Intuitive Robot Teleoperation Through Multi-Sensor Informed Mixed Reality Visual Aids

Abstract

The work carried out by the University of Catania is in the framework of the project "Safe and Smart Farming with Artificial Intelligence and Robotics-programma ricerca di ateneo UNICT 2020-22 linea 2" ABSTRACT Mobile robotic systems have evolved to include sensors capable of truthfully describing robot status and operating environment as accurately and reliably as never before. This possibility is challenged by effective sensor data exploitation, because of the cognitive load an operator is exposed to, due to the large amount of data and time-dependency constraints. This paper addresses this challenge in remote-vehicle teleoperation by proposing an intuitive way to present sensor data to users by means of using mixed reality and visual aids within the user interface. We propose a method for organizing information presentation and a set of visual aids to facilitate visual communication of data in teleoperation control panels. The resulting sensor-information presentation appears coherent and intuitive, making it easier for an operator to catch and comprehend information meaning. This increases situational awareness and speeds up decision-making. Our method is implemented on a real mobile robotic system operating outdoor equipped with on-board internal and external sensors, GPS, and a reconstructed 3D graphical model provided by an assistant drone. Experimentation verified feasibility while intuitive and comprehensive visual communication was confirmed through an assessment, which encourages further developments.

Illustration Art Based on Visual Communication in Digital Context

“…From the results of 3D reconstruction accuracy of different images, we can see that the 3D image virtual reconstruction system designed in this paper is almost the same as the stereo vision system in references [17,20]. However, in terms of user satisfaction, the system in this paper has more advantages, because it effectively uses the design principle of visual communication effect.…”

“…is is mainly because the particle swarm optimization algorithm applied in this method has nothing to do with the size of its neighbors, which reduces the number of iterations, reduces the complexity of the algorithm, and realizes the fast visual communication of images of any size. In addition, in order to carry out experimental comparative analysis, the system of this paper and the stereo vision system proposed in literature [17] and literature [20] are used to reconstruct the illustration images in three dimensions, and the reconstruction accuracy and user satisfaction are evaluated. e results are as follows: Figures 6 and 7.…”

“…is method uses the mathematical model of retinal neuron recognition to preprocess the image and detect dim targets. Reference [20] proposed an adaptive 3D reconstruction algorithm of light field based on array images. References [21,22] used wavelet to decompose the image twice to obtain the high-frequency component features of the object.…”

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

“…Figure : [7] -Human-vehicle interaction projected guidance [2, 7], interaction with pedestrians [64,88,320], display for passengers [223] Augmented Wheelchair projecting intentions [458], displaying virtual hands to convey intentions [300], self-navigating wheelchair [314], assistive features [495] Navigation tangible 3D map [258], automobile navigation [438] Search and Rescue 35 Figure: [363] -Ground search collaborative ground search [296,343,360,362,363], target detection and notification [211,241,266,269,305,361,468,479], teleoperated ground search [54,73,102,146,156,234,237,238,267,268,365,380,417,469,483,484,486] Aerial search drone-assisted search and rescue [114,332,482], target detection and highlight…”

Towards a balancing safety against performance approach in human–robot co-manipulation for door-closing emergencies

“…Uncertain fire conditions and the excessive use of protective equipment further increase the challenges placed on the balance control system. Although existing telemanipulation is a highly efficient master-slave work pattern because of human-in-the-loop control, it is not trivial for the human operators to specify the optimal measures to guarantee robotic safety [3][4][5].…”