Mirrorlabs - creating accessible Digital Twins of robotic production environment with Mixed Reality

Abstract

How to visualize recorded production data in Virtual Reality? How to use state of the art Augmented Reality displays that can show robot data? This paper introduces an opensource ICT framework approach for combining Unity-based Mixed Reality applications with robotic production equipment using ROS Industrial. This publication gives details on the implementation and demonstrates the use as a data analysis tool in the context of scientific exchange within the area of Mixed Reality enabled human-robot co-production.

Simulation Components in Gazebo

“…In addition, A MR study on human behavior based on a Gazebo simulation is presented in [8]. The MirrorLabs project features the development of MR applications primarily for educational purposes [9]. Our work continues ideas sketched in [10] where we evaluated latency issues in distributed Gazebo usage scenarios.…”

Framework for Armature-Based 3D Shape Reconstruction of Sensorized Soft Robots in eXtended Reality

“…The implementation of XR in manufacturing has been proven feasible, such as AR-based repair instructions ( Aschenbrenner et al, 2019 ) or displaying additional information within a factory ( Peake et al, 2016 ). While significant research was conducted for each topic, HRC and XR, cross-technology research is yet less prominent with very few exceptions ( Aschenbrenner et al, 2020 ). With the increasing popularity of using digital twins, XR can serve as a way to interface with a digital twin in the context of its intended physical environment.…”

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

“…Figure 10 suggests that there are less explored application domains, which can be investigated in the future, which include design and creative tasks, remote collaboration, and workspace applications. Manufacturing: joint assembly [21,30,43,138,289], grasping and manipulation [ [67,79,153,185,379], tutorial and simulation [52], welding [313] Maintenance: maintenance of robots [144,252], remote repair [31,48], performance monitoring [126,128], setup and calibration [352], debugging [373] Safety and Inspection: nuclear detection [15], drone monitoring [76,114,171,482], safety feature [44,66,104,379,385], ground monitoring [211,269,479] Automation and Teleoperation: interactive programming interface [118,131,136,177,270,288,324,496] Logistics: package delivery [265] Aerospace: surface exploration [54], teleoperated manipulator…”

“…Workshop [44,45,167,244,246,274,476], Theoretical Framework [200,374], Example Applications [30,47,66,84,124,127,142,145,159,174,195,197,209,210,243,245,252,265,269,315,325,341,342,366,368,382,385,418,425,431,476,481,483,496], Focus Groups [272,467], Early Demonstrations [24, 49, 53, 55, 82, 89, 96, 102, 104, 111, 115, 117, 136, 149, 162, 199, 216, 221, 239-241, 251, 255, 257, 270, 290, 305, 338, 346, 370, 375, 387, 400, 404, 408, 414, 436, 451, 479], Case Studies [65,105,…”

“…• Domestic and everyday use (32) • Industry (122) • Entertainment (30) • Education and training (15) • Social interaction (18) • Design and creative tasks (10) • Medical and health (15) • Remote collaboration (6) • Mobility and transportation (11) • Search and rescue (20) • Workspace and knowledge work (9) • Data physicalization (12)…”