Matterport3D: Learning from RGB-D Data in Indoor Environments

Chang, Angel X.; Dai, Angela; Funkhouser, Thomas; Halber, Maciej; Nießner, Matthias; Savva, Manolis; Song, Shuran; Zeng, Andy; Zhang, Yinda

doi:10.48550/arxiv.1709.06158

Cited by 219 publications

(263 citation statements)

References 36 publications

Supporting

Mentioning

260

Contrasting

Order By: Relevance

“…A real-time simulation environment is used to simulate the dynamics and obstacle avoidance behavior of the robots involved in this work. The simulation is implemented in the Habitat-AI environment, a photo-realistic, physics-based simulator, to ensure that the framework and test results are transferable to real-world deployment [35][36][37] We choose a scene from the Matterport3D dataset, which is a collection of 90 scans, to design a sample tour of an indoor environment. Though the simulated environment is a house, it contains most of the features that would exist in a museum.…”

Section: Habitat-ai Simulation Environmentmentioning

confidence: 99%

Simultaneous Human-robot Matching and Routing for Multi-robot Tour Guiding under Time Uncertainty

Fu,

Kathuria,

Rizzo

et al. 2022

Preprint

View full text Add to dashboard Cite

This work presents a framework for multi-robot tour guidance in a partially known environment with uncertainty, such as a museum. In the proposed centralized multi-robot planner, a simultaneous matching and routing problem (SMRP) is formulated to match the humans with robot guides according to their selected places of interest (POIs) and generate the routes and schedules for the robots according to uncertain spatial and time estimation. A large neighborhood search algorithm is developed to efficiently find sub-optimal lowcost solutions for the SMRP. The scalability and optimality of the multi-robot planner are evaluated computationally under different numbers of humans, robots, and POIs. The largest case tested involves 50 robots, 250 humans, and 50 POIs. Then, a photo-realistic multi-robot simulation platform was developed based on Habitat-AI to verify the tour guiding performance in an uncertain indoor environment. Results demonstrate that the proposed centralized tour planner is scalable, makes a smooth trade-off in the plans under different environmental constraints, and can lead to robust performance with inaccurate uncertainty estimations (within a certain margin).

show abstract

Section: Habitat-ai Simulation Environmentmentioning

confidence: 99%

Simultaneous Human-robot Matching and Routing for Multi-robot Tour Guiding under Time Uncertainty

Fu,

Kathuria,

Rizzo

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To you, this single image represents a rich 3D world in which the cabinet continues behind the chairs. This work aims to learn a mapping from a single image to a 3D reconstruction of a scene, including visible and occluded surfaces, while learning from real, unstructured scans like Matterport3D [5] or ScanNet [10]. These non-watertight scans are currently one of the richest sources of real-world 3D ground truth, and as more devices integrate LIDAR scanners, their importance will only grow.…”

Section: Introductionmentioning

confidence: 99%

“…enables using signed distance functions (SDF) or occupancy functions, but limits methods to watertight data like ShapeNet [6], humans [40], or memorizing single watertight scenes [45]. Real 3D scans (e.g., [5,10]), on the other hand are off-limits. Exceptions include [8], which fits an unsigned distance function (UDF) to instance-specific models, and SAL [1,2] which learns SDFs on objects with welldefined insides and outsides that also have holes.…”

Section: Introductionmentioning

confidence: 99%

“…We believe the lack of success on learning to predict implicit functions on datasets like Matterport3D [5] stems from three key challenges. First, the unstructured, scenelevel nature of the 3D data precludes functions like the SDF or occupancy: even if one starts with watertight surfaces, clipping with the view frustum can make the SDF poorly defined.…”

Section: Introductionmentioning

confidence: 99%

“…We compare our proposed approach, DRDF, with other distance functions and ways of obtaining a reconstruction of the scene, such as Layered Depth Images (LDI) [44]. Our experiments ( §4) compare the approaches on Matterport3D [5], 3DFront [17], and ScanNet [10]. Our evaluations show that the DRDF is substantially better at recovering the full 3D scene across a variety of metrics, including both visible and invisible regions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction

Kulkarni¹,

Johnson²,

Fouhey³

2021

Preprint

View full text Add to dashboard Cite

We present an approach for scene-level 3D reconstruction, including occluded regions, from an unseen RGB image. Our approach is trained on real 3D scans and images. This problem has proved difficult for multiple reasons; Real scans are not watertight, precluding many methods; distances in scenes require reasoning across objects (making it even harder); and, as we show, uncertainty about surface locations motivates networks to produce outputs that lack basic distance function properties. We propose a new distance-like function that can be computed on unstructured scans and has good behavior under uncertainty about surface location. Computing this function over rays reduces the complexity further. We train a deep network to predict this function and show it outperforms other methods on Matterport3D, 3D Front, and ScanNet.https://nileshkulkarni.github.io/scene_drdf

show abstract

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans

Liu

Furukawa

2018

Lecture Notes in Computer Science

125

114

View full text Add to dashboard Cite

The ultimate goal of this indoor mapping research is to automatically reconstruct a floorplan simply by walking through a house with a smartphone in a pocket. This paper tackles this problem by proposing FloorNet, a novel deep neural architecture. The challenge lies in the processing of RGBD streams spanning a large 3D space. FloorNet effectively processes the data through three neural network branches: 1) PointNet with 3D points, exploiting the 3D information; 2) CNN with a 2D point density image in a top-down view, enhancing the local spatial reasoning; and 3) CNN with RGB images, utilizing the full image information. FloorNet exchanges intermediate features across the branches to exploit the best of all the architectures. We have created a benchmark for floorplan reconstruction by acquiring RGBD video streams for 155 residential houses or apartments with Google Tango phones and annotating complete floorplan information. Our qualitative and quantitative evaluations demonstrate that the fusion of three branches effectively improves the reconstruction quality. We hope that the paper together with the benchmark will be an important step towards solving a challenging vector-graphics reconstruction problem. Code and data are available at https://github.com/art-programmer/FloorNet.

show abstract

Matterport3D: Learning from RGB-D Data in Indoor Environments

Cited by 219 publications

References 36 publications

Simultaneous Human-robot Matching and Routing for Multi-robot Tour Guiding under Time Uncertainty

Simultaneous Human-robot Matching and Routing for Multi-robot Tour Guiding under Time Uncertainty

What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans

Contact Info

Product

Resources

About