Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

Abstract

In hazardous environments, where conditions present risks for humans, the maintenance and interventions are often done with teleoperated remote systems or mobile robotic manipulators to avoid human exposure to dangers. The increasing need for safe and efficient teleoperation requires advanced environmental awareness and collision avoidance. The up-to-date screen-based 2D or 3D interfaces do not fully allow the operator to immerse in the controlled scenario. This problem can be addressed with the emerging Mixed Reality (MR) technologies with Head-Mounted Devices (HMDs) that offer stereoscopic immersion and interaction with virtual objects. Such human-robot interfaces have not yet been demonstrated in telerobotic interventions in particle physics accelerators. Moreover, robotic operations often require a few experts to collaborate, which increases the system complexity and requires sharing a multi-user Augmented Reality (AR) workspace. The multi-user telerobotics with shared control in the AR has not yet been approached in the industrial state-of-the-art. In this work, the developed MR human-robot interface using the AR HMD is presented. The interface adapts to the constrained wireless networks in particle accelerator facilities and provides reliable high-precision interaction and specialized visualization. The multimodal operation uses hands, eyes and user motion tracking, and voice recognition for control, as well as offers video, 3D point cloud and audio feedback from the robot. Multiple experts can collaborate in the AR workspace locally or remotely, share the robot's control and monitor robotic teleoperation. Ten (10) operators tested the interface in intervention scenarios in the European Organization for Nuclear Research (CERN) with complete network characterization and measurements to conclude if operational requirements were met and if the network architecture could support single and multi-user communication load.The interface system has proved to be operationally ready at the Technical Readiness Level (TRL) 8 -and was validated through successful tests and demonstration in single and multi-user missions. Some areas of system limitations and further work were identified, such as optimising the network architecture for multi-user scenarios or high-level interface actions applying automatic interaction strategies with the robot depending on network conditions.

Saliency-Guided Point Cloud Compression for 3D Live Reconstruction

“…Nowadays, the 3D representation of objects and environments is increasingly utilized in common applications like telepresence and remote collaboration [1,2], autonomous driving [3], remote operations in hazardous environments [4,5], telemedicine [6,7]. Specifically, thanks to the use of special sensors such as depth (RGB-D) cameras or LiDAR scanners, it is possible to faithfully reconstruct reality and reproduce it in virtual environments, enabling new forms of interaction.…”

Robotics via Experiential Active Learning With Immersive Technology: A Pedagogical Framework Designed for Engineering Curricula

“…It is worth noting that haptic interface could also be included in the extension of reality other than the usual visual interface. There have been recent efforts to incorporate immersive technology in robotic applications [10], [11], especially in the subject of human-robot interaction (HRI) [12], [13] in the context of Industry 4.0 [14], [15], surgical procedure [16], [17] and micro-manipulation [17], [18]. Assimilating immersive technology for robot applications has shown promising potential in enhancing human-robot interaction and intuitive machine operation via XR interfaces.…”

Mixed Reality Human-Robot Interface for Robotic Operations in Hazardous Environments

“…In most cases, each functionality can be used multimodally with 2-4 input types at the operator's convenience (hands near or far interaction, hand gestures and tracking, voice command, eyes tracking + dwell/voice command). Each functionality has a graphical example if the interface implements it, which can be consulted in its corresponding figure or the video demonstration available in [65].…”

“…The actions performed by the controlling operator are synchronized in the display of the monitoring operator. The video demonstration [65] shows the process and collaboration between operators.…”

“…Similar to the base control (explained in Figure 19, there is the double confirmation system, and the manipulator's end-effector moves and rotates at a speed proportional to the hand displacement and rotation. In the video demonstration [65], it is presented how the robot is operated this way. 3) In the Planning Forward And Backward Reaching Inverse Kinematics (FABRIK) mode, the end-effector target position is moved with hand near or far interaction (Figure 24).…”