Gibson Env: Real-World Perception for Embodied Agents

Abstract

Figure 1: Two agents in Gibson Environment for real-world perception. The agent is active, embodied, and subject to constraints of physics and space (a,b). It receives a constant stream of visual observations as if it had an on-board camera (c). It can also receive additional modalities, e.g. depth, semantic labels, or normals (d,e,f). The visual observations are from real-world rather than an artificially designed space. AbstractDeveloping visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment 1 for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments

“…This advocates the use of simulation. Our previous work, the Gibson Environment [8], provided a simulation environment to train embodied agents on visual navigation tasks without interactions. The main advantage of Gibson V1 is that it generates photo-realistic virtual images for the agent.…”

“…Gibson V1 [8] provides a massive dataset of high quality reconstructions of indoor scenes. However, each reconstruction consists of a single static mesh, which does not afford interaction or changes in the configuration of the environment ( Fig.…”

SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation

“…Effective policies generalize by understanding the geometry of the scenes rather than trying to localize into a known map based on the visual inputs. SplitNet outperforms all other methods by a wide margin on all three environments SUNCG [30] MP3D [3] Gibson [35] SPL Success SPL Success SPL Success Table 1. Performance on Unseen Environments.…”

Situational Fusion of Visual Representation for Visual Navigation

“…In this work, we also use simulated data to train our models. However, our simulator of choice, Gibson [37], employs 3D captures of real-world scenes and domain-adapted simulation, which has been shown effective in deploying the simulation-trained policies directly in the real world [17,24]. We focus on improving the generalization of our models across visual scenes with novel ways of fusing the diverse set of visual representations.…”