Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

Abstract

Figure 1: The Habitat-Matterport 3D (HM3D) dataset of large-scale 3D and photorealistic environments provides 1,000 building-scale reconstructions of interiors from a diverse set of geographic locations. The scale, completeness, and visual fidelity of these reconstructions surpass those of prior datasets, and enable research on embodied AI agents that can perceive, navigate, and act within realistic indoor environments. The image on the left displays a collage of a subset of HM3D scans. The image on the top-right is a close-up view of a specific scan, and the images on the bottom-right are snapshots from two camera viewpoints in the scan.

Boosting Reinforcement Learning with Unsupervised Feature Extraction

“…We use the Habitat 2.0 environment [10,12] combined with the photo realistic Habitat-Matterport 3D dataset [8] (Fig. 1).…”

Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation

“…Dataset. For training and evaluation, we use a combination of the Habitat-Matterport (HM3D) [13] and Gibson [51] 3D datasets. The two datasets combined consist of over 1000 high-resolution 3D scans of real-world indoors environments, and consists of realistic clutter.…”

“…The sim2real paradigm consists of training robots in simulation (potentially for billions of simulation steps corresponding to decades of experience [1]) before deploying them in reality. The last few years have seen significant investments -the development of new simulators [2][3][4][5][6][7][8][9][10][11][12], curation and annotation of 3D scans and assets [13][14][15], and development of techniques for overcoming the sim2real gap [16][17][18][19] -resulting in a number of successful demonstrations of sim2real transfer [20][21][22][23][24][25]. However, no simulator is a perfect replica of reality and the main challenge in this paradigm is overcoming the sim2real gap, defined as the drop in a robot's performance in the real-world (compared to simulation).…”

Skill Fusion in Hybrid Robotic Framework for Visual Object Goal Navigation

“…That requires a fast-performing simulator that should also be photo-realistic to be able to transfer the resulting policy to the real world. To this end, we used the fastest photorealistic simulator BPS [3] with the largest 1000-scene dataset, HM3D [43]. To train the RL Exploration skill, we take the train part of HM3D (800 scenes) and 145 scenes for the RL GoalReacher skill, as HM3D has only that number with available ground truth semantics.…”