RoutedFusion: Learning Real-time Depth Map Fusion

Abstract

The efficient fusion of depth maps is a key part of most state-of-the-art 3D reconstruction methods. Besides requiring high accuracy, these depth fusion methods need to be scalable and real-time capable. To this end, we present a novel real-time capable machine learning-based method for depth map fusion. Similar to the seminal depth map fusion approach by Curless and Levoy, we only update a local group of voxels to ensure real-time capability. Instead of a simple linear fusion of depth information, we propose a neural network that predicts non-linear updates to better account for typical fusion errors. Our network is composed of a 2D depth routing network and a 3D depth fusion network which efficiently handle sensor-specific noise and outliers. This is especially useful for surface edges and thin objects for which the original approach suffers from thickening artifacts. Our method outperforms the traditional fusion approach and related learned approaches on both synthetic and real data. We demonstrate the performance of our method in reconstructing fine geometric details from noise and outlier contaminated data on various scenes.

How good MVSNets are at depth fusion

“…For both methods we used the original PyTorch implementations 3 . For generation of the final point clouds we also used the codes from the original repositories of the methods.…”

“…Although there exist impressive methods combining multi-view photo and depth fusion, such as [2], to the best of our knowledge, there is no such method based on the deep learning approach. At the same time, deep learning has been recently demonstrated to be beneficial for both depth fusion and MVS, see, e.g., [3] and Section 2.…”

Atlas: End-to-End 3D Scene Reconstruction from Posed Images

“…3D-SIC [20] focuses on 3D instance segmentation using region proposals and adds a per instance completion head. Routed fusion [47] uses 2D filtering and 3D convolutions in view frustums to improve aggregation of depth maps.…”

Multi-Resolution 3D Mapping with Explicit Free Space Representation for Fast and Accurate Mobile Robot Motion Planning

“…Another strategy is to exploit memory-efficient data structures, such as octree-based voxel grids [4], [12], [13] or hash tables [14], [15], for quicker spatial indexing. More recently, deep learning methods tackling this problem in an incremental fashion are also being introduced [16]. Despite this impressive progress, most research in 3D mapping has targeted the application of surface reconstruction, in which the objective is to produce a high-quality mesh/point-cloud of a given scene.…”