Mixed Reality-Enhanced Intuitive Teleoperation with Hybrid Virtual Fixtures for Intelligent Robotic Welding

Abstract

This paper presents an integrated scheme based on a mixed reality (MR) and haptic feedback approach for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed robotic tele-welding system features imitative motion mapping from the user’s hand movements to the welding robot motions, and it enables the spatial velocity-based control of the robot tool center point (TCP). The proposed mixed reality virtual fixture (MRVF) integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Onsite welding and tele-welding experiments identify the operational differences between professional and unskilled welders and demonstrate the effectiveness of the proposed MRVF tele-welding framework for novice welders. The MRVF-integrated visual/haptic tele-welding scheme reduced the torch alignment times by 56% and 60% compared to the MRnoVF and baseline cases, with minimized cognitive workload and optimal usability. The MRVF scheme effectively stabilized welders’ hand movements and eliminated undesirable collisions while generating smooth welds.

Effectiveness of IoT and VR for Real-Time Teleoperation of Industrial Robots

“…With the advantage of virtual reality, teleoperation robot platforms have become particularly useful for hazardous and obstructed environments, allowing digital transformation and improved scalability. Yunpeng Su et al [12] proposed virtual reality platform with haptic input device and Hidden Markov models algorithm to improve performance and intuitiveness in remote welding operation. Yuan Sun et al implement VR [62] to support construction-related tasks with mobile robots.…”

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

“…By implementing MR as a human-robot interaction interface (HRI), the user experiences a physical presence in the remote environment and co-existence with the robotic platform via MR subspace, while guiding and monitoring the robotic platform at the local user space [105,106]. MR interfaces blend virtual elements with the real world, providing operators with an immersive 3D visualization of the remote environment, such as a welding site [107]. This immersion allows operators to perceive the task space in depth, enhancing their understanding of the work area and minimizing cognitive load.…”

Master-Slave Arm Heterogeneous Mapping With Link Pose Constraint in Teleoperation System

“…The expression of θ3 in the inverse solution of the robot is shown below: Equation (13). According to the forward kinematics of the robot wrist, the position (p rw x , p rw y , p rw z ) of the robot wrist joint can be expressed as follows: Let the element of Equation ( 19) be presented in conjunction with the robot model (20) This means that the robot wrist joint is out of its reachable space when 2 2 2 3 4 0 a d K    .…”

“…Wu et al [9] proposed a vision-based human-robot teleoperation system based on monocular human pose estimation convolutional neural networks. In addition, virtual reality, mixed reality, and augmented reality technologies are applied to heterogeneous teleoperation systems to improve the intuitiveness and flexibility of the teleoperation systems [10]- [13]. Among these heterogeneous teleoperation systems, the key is to realize effective pose mapping between master and slave arms.…”