UnrealROX: an extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation

Abstract

Data-driven algorithms have surpassed traditional techniques in almost every aspect in robotic vision problems. Such algorithms need vast amounts of quality data to be able to work properly after their training process. Gathering and annotating that sheer amount of data in the real world is a time-consuming and error-prone task. Those problems limit scale and quality. Synthetic data generation has become increasingly popular since it is faster to generate and automatic to annotate. However, most of the current datasets and environments lack realism, interactions, and details from the real world. UnrealROX is an environment built over Unreal Engine 4 which aims to reduce that reality gap by leveraging hyperrealistic indoor scenes that are explored by robot agents which also interact with objects in a visually realistic manner in that simulated world. Photorealistic scenes and robots are rendered by Unreal Engine into a virtual reality headset which captures gaze so that a human operator can move the robot and use controllers for the robotic hands; scene information is dumped on a per-frame basis so that it can be reproduced offline to generate raw data and ground truth annotations. This virtual reality environment enables robotic vision researchers to generate realistic and visually plausible data with full ground truth for a wide variety of problems such as class and instance semantic segmentation, object detection, depth estimation, visual grasping, and navigation.

Evaluation of different chrominance models in the detection and reconstruction of faces and hands using the growing neural gas network

“…In order to address some of the challenges we face in real environments with changes in illumination, shadows and more cluttered backgrounds, we have tested the different chrominance models on two synthetic images we have generated with our virtual environment UnrealROX [32]. Figure 7 shows the segmentation of the skin colour after applying the four different chrominance models and getting the skin probability with a GMM.…”

“…Figures 9 and 10 show the whole process for different input samples that were synthetically generated using UnrealROX [32]. From left to right, we can see the original RGB image and its corresponding ground truth segmentation mask.…”

“…Image segmentation on two synthetic images generated with UnrealROX [32]. From top to bottom, original input image, and probability maps for the skin colour after applying the four different chrominance models…”

The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions

“…To record and generate all the data for this dataset, we made extensive use of a tool that was specifically built for this dataset: UnrealROX [30], a virtual reality environment for generating synthetic data for various robotic vision tasks. In such environment, a human operator can be embodied, in virtual reality, as a robot agent inside a scene to freely navigate and interact with objects as if it was a realworld robot.…”

A novel robotic framework for safe inspection and telemanipulation in hazardous and unstructured environments

“…Applying the imitation leaning [147] the robot is able to learn and perform complex tasks or solve the complex dexterous manipulation problem [143]. By using external controller or human body tracking, bimanual robot is easily controlled [383][384] [385]; this last one was presented in DARPA Robotics Chalenge [386] performing complex manipulation tasks. Moreover, it is possible to add real pictures features as texture of the virtual environment [387] and other feedback source in order to increase of realism, help to go out from the gaming effect and decrease the fatigue and stress while using VR [388].…”