From 2D to 3D Mixed Reality Human-Robot Interface in Hazardous Robotic Interventions with the Use of Redundant Mobile Manipulator

Szczurek, Krzysztof Adam; Marı́n, R.; Matheson, Eloise; Perier, Hugo; Buonocore, Luca Rosario; Castro, Mario Di

doi:10.5220/0010528503880395

Cited by 3 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Involving Mixed Reality in the robotic manipulator programming, as presented in [11], indicated the benefits of reducing program writing time and the number of errors due to virtual simulation in a virtual environment. Also, during a preliminary study and a 3D MR interface pilot project [12] at CERN, it was concluded that a transition from a 2D-based to 3D-based interfaces could bring multiple benefits, such as better spatial cognition, collisions avoidance or detection, motion planning, situational awareness, three-dimensional perception of the environment and spatial vision cues. Which, as a result, increased the capabilities and safety of teleoperation.…”

Section: A Transition From 2d To 3d Mixed Reality Human-robot Interfacesmentioning

confidence: 99%

“…Therefore, an eye-tracking system has been implemented in the humanrobot interface [36], which prevented dangerous collisions by constraining the robot's movements and decreasing its speed when the operator became distracted or was not observing the robot's video feedback. A pilot project studying the benefits of the transition to the 3D MR interface and preliminary operator vital parameters monitoring assessment was conducted in [12], which motivated the further work presented here.…”

Section: Vital Parameters Monitoring In Hazardous Environments At Cernmentioning

confidence: 99%

“…The 9 DOF robotic manipulator is installed on the TIM that operates in the LHC. In this picture [12], the arm is in the deployed state. A radioactive source is placed in the end-effector, and the arm must reach the BLM at a specified distance to verify the correct functioning of the device.…”

Section: Motivation and Problem Formulationmentioning

confidence: 99%

“…The manipulator's task is to approach a BLM while holding a radioactive source at a specified distance. More explanation of the scenario, robot and task can be found in [12] in Section 1.1 and Section IE of [43].…”

Section: A Controlled Robotmentioning

confidence: 99%

“…The tasks reflected real tasks that operators faced during BLM robotic measurements campaign [12]. There were two types of tasks: and b) and were calibrated individually accordingly to a participants' baselines (i.e.…”

Section: A Tasks and Interfaces Usementioning

confidence: 99%

See 4 more Smart Citations

Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

et al. 2023

View full text Add to dashboard Cite

Remote robotic interventions and maintenance tasks are frequently required in hazardous environments. Particularly, missions with a redundant mobile manipulator in the world's most complex machine, the CERN Large Hadron Collider, are performed in a sensitive underground environment with radioactive or electromagnetic hazards, bringing further challenges in safety and reliability. The mission's success depends on the robot's hardware and software, and when the tasks become too unpredictable to execute autonomously, the operators need to make critical decisions. However, in most current humanmachine systems, the state of the human is neglected. In this context, a novel 3D Mixed Reality (MR) human-robot interface with the Operator Monitoring System (OMS) was developed to advance safety and task efficiency with improved spatial awareness, advanced manipulator control, and collision avoidance. However, new techniques could increase the system's sophistication and add to the operator's workload and stress. Therefore, for operational validation, the 3D MR interface had to be compared with an operational 2D interface, which has been used in hundreds of interventions. With the 3D MR interface, the execution of precise approach tasks was faster, with no increased workload or physiological response. The new 3D MR techniques improved the teleoperation quality and safety while maintaining similar effects on the operator. The OMS worked jointly with the interface and performed well with operators with varied teleoperation backgrounds facing a stressful real telerobotic scenario in the LHC. The paper contributes to the methodology for human-centred interface evaluation incorporating the user's physiological state: heart rate, respiration rate and skin electrodermal activity, and combines it with the NASA TLX assessment method, questionnaires, and task execution time. It provides novel approaches to operator state identification, the GUI-OMS software architecture, and the evaluation of the 3D MR techniques. The solutions can be practically applied in mission-critical applications, such as telesurgery, space robotics, uncrewed transport vehicles and semi-autonomous machinery.

show abstract

Section: A Transition From 2d To 3d Mixed Reality Human-robot Interfacesmentioning

confidence: 99%

Section: Vital Parameters Monitoring In Hazardous Environments At Cernmentioning

confidence: 99%

Section: Motivation and Problem Formulationmentioning

confidence: 99%

Section: A Controlled Robotmentioning

confidence: 99%

Section: A Tasks and Interfaces Usementioning

confidence: 99%

See 3 more Smart Citations

Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

et al. 2023

View full text Add to dashboard Cite

show abstract

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

Su,

Chen,

Zhou

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

show abstract

Mixed Reality Human-Robot Interface for Robotic Operations in Hazardous Environments

Szczurek

View full text Add to dashboard Cite

vi y el estrés del operador al tiempo que aumentaba la complejidad del sistema. La MR 3D del CERN y las interfaces 2D operativas se compararon utilizando el método de evaluación TLX de la NASA, cuestionarios personalizados, curvas de tiempo de ejecución de tareas y mediante la medición de la frecuencia cardíaca (HR), la frecuencia respiratoria (RR) y la actividad electrodérmica de la piel ( EDA) evaluado por el Sistema de Monitorización del Operador (OMS) desarrollado. El sistema fue diseñado para medir los parámetros fisiológicos de un teleoperador durante intervenciones robóticas.Se identificaron limitaciones y otras áreas de investigación para mejorar, como la optimización de la arquitectura de red para escenarios multiusuario y la aplicación de estrategias de interacción automática según las condiciones de la red para acciones de interfaz de nivel superior. El uso práctico de OMS reveló la necesidad de aplicar técnicas de aprendizaje automático en la interpretación de señales para detectar situaciones no estándar y utilizar tecnología de monitorización sin contacto. Los sistemas de interfaz desarrollados demostraron preparación operativa, logrando un nivel de preparación técnica (TRL) 8, a través de misiones exitosas de uno o varios usuarios. vii

show abstract

From 2D to 3D Mixed Reality Human-Robot Interface in Hazardous Robotic Interventions with the Use of Redundant Mobile Manipulator

Cited by 3 publications

References 0 publications

Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

Mixed Reality Human-Robot Interface for Robotic Operations in Hazardous Environments

Contact Info

Product

Resources

About