A Novel Framework for Mixed Reality–Based Control of Collaborative Robot: Development Study

Abstract

Background Applications of robotics in daily life are becoming essential by creating new possibilities in different fields, especially in the collaborative environment. The potentials of collaborative robots are tremendous as they can work in the same workspace as humans. A framework employing a top-notch technology for collaborative robots will surely be worthwhile for further research. Objective This study aims to present the development of a novel framework for the collaborative robot using mixed reality. Methods The framework uses Unity and Unity Hub as a cross-platform gaming engine and project management tool to design the mixed reality interface and digital twin. It also uses the Windows Mixed Reality platform to show digital materials on holographic display and the Azure mixed reality services to capture and expose digital information. Eventually, it uses a holographic device (HoloLens 2) to execute the mixed reality–based collaborative system. Results A thorough experiment was conducted to validate the novel framework for mixed reality–based control of a collaborative robot. This framework was successfully applied to implement a collaborative system using a 5–degree of freedom robot (xArm-5) in a mixed reality environment. The framework was stable and worked smoothly throughout the collaborative session. Due to the distributed nature of cloud applications, there is a negligible latency between giving a command and the execution of the physical collaborative robot. Conclusions Opportunities for collaborative robots in telerehabilitation and teleoperation are vital as in any other field. The proposed framework was successfully applied in a collaborative session, and it can also be applied in other similar potential applications for robust and more promising performance.

Security Vulnerabilities and Threats in Robotic Systems: A Comprehensive Review

“…One of the main benefits of the Robot Framework is its flexibility in test case creation, as it allows collaboration between technical and non-technical team members. The framework's keyword-driven approach promotes test case reusability, simplifying maintenance and scalability [51].…”

Hand Rehabilitation Devices: A Comprehensive Systematic Review

“…Telerehabilitation offers the perfect balance of at-home solutions with the care of a professional. Telerehabilitation effectively provides the necessary exercise for patients [ 101 , 102 ]. Furthermore, in combination with VR or exercise games, telerehabilitation yielded high satisfaction from the patients who use the system [ 103 , 104 ].…”

Augmented reality‐based telepresence in a robotic manipulation task: An experimental evaluation

“…Ponomareva et al [30] demonstrated a system for teleoperated robots in medical scenarios, which included an augmented virtual environment and region-based convolutional neural network for identifying and positioning laboratory instruments. Finally, Shahria et al [31] explored AR-based teleoperation, Chan et al [32] used AR in shared manufacturing tasks, and Brizzi et al [33] focused on AR for embodiment where the robot perspective is reconstructed. AR has also been used in robot programming, the creation of training scenarios, and the development of physical artefacts, enabling the possibility of programming a robot in a workspace without its physical presence and simulating motions before their execution [34][35][36][37].…”

“…Finally, Shahria et al. [31] explored AR‐based teleoperation, Chan et al. [32] used AR in shared manufacturing tasks, and Brizzi et al.…”